{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 06_Train_DistMult\n",
    "#\n",
    "# created by LuYF-Lemon-love <luyanfeng_nlp@qq.com> on February 27, 2023\n",
    "# updated by LuYF-Lemon-love <luyanfeng_nlp@qq.com> on February 27, 2023\n",
    "#\n",
    "# 该脚本展示了如何在 DRKG 上训练模型 (DistMult), 并利用网格搜索寻找到最优参数.\n",
    "#\n",
    "# 需要的包:\n",
    "#          torch\n",
    "#          dgl, version: 0.4.3\n",
    "#          dglke\n",
    "#          numpy\n",
    "#\n",
    "# 需要的文件:\n",
    "#          ./dataset\n",
    "#\n",
    "# 源教程链接: https://github.com/gnn4dr/DRKG/blob/master/embedding_analysis/Train_embeddings.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training DRKG Using DistMult\n",
    "\n",
    "这个 notebook 展示了如何在 DRKG 上训练模型 (DistMult), 并利用网格搜索寻找到最优参数."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入需要的库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 网格搜索参数\n",
    "\n",
    "我们能使用 DGL-KE 命令训练 DistMult 模型, 关于如何使用 DGL-KE 的更多信息请参考 https://github.com/awslabs/dgl-ke.\n",
    "\n",
    "这里我们使用两个 GPU 训练模型."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: **200**, 400\n",
    "\n",
    "- gamma: **50**, 125, 200\n",
    "\n",
    "- lr: **0.01**, 0.05, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.312 seconds\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.42179756767898796\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.4218365873351693\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.6085648582309484\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.6083566954851151\n",
      "[proc 1][Train](20000/100000) average loss: 0.5151812129944563\n",
      "[proc 0][Train](20000/100000) average loss: 0.5150966415151954\n",
      "[proc 1][Train](20000/100000) average regularization: 0.020644050214713206\n",
      "[proc 1][Train] 20000 steps take 355.567 seconds\n",
      "[proc 1]sample: 64.617, forward: 152.108, backward: 48.961, update: 83.055\n",
      "[proc 0][Train](20000/100000) average regularization: 0.020649947847187287\n",
      "[proc 0][Train] 20000 steps take 355.571 seconds\n",
      "[proc 0]sample: 66.213, forward: 151.228, backward: 48.800, update: 88.628\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.41659595900475976\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.4163659319266677\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.5818226113408804\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.5819617623701692\n",
      "[proc 0][Train](40000/100000) average loss: 0.49909427175819876\n",
      "[proc 1][Train](40000/100000) average loss: 0.49927886078357697\n",
      "[proc 0][Train](40000/100000) average regularization: 0.023133664725441486\n",
      "[proc 0][Train] 20000 steps take 353.099 seconds\n",
      "[proc 0]sample: 65.159, forward: 141.501, backward: 46.861, update: 90.173\n",
      "[proc 1][Train](40000/100000) average regularization: 0.023131319496035575\n",
      "[proc 1][Train] 20000 steps take 353.108 seconds\n",
      "[proc 1]sample: 67.057, forward: 150.632, backward: 49.453, update: 84.872\n",
      "[0]Valid average MRR: 0.24803044343604166\n",
      "[0]Valid average MR: 173.1802854496922\n",
      "[0]Valid average HITS@1: 0.17619641008879447\n",
      "[0]Valid average HITS@3: 0.2667613172086942\n",
      "[0]Valid average HITS@10: 0.3785442882823991\n",
      "[proc 0]validation take 373.315 seconds:\n",
      "[1]Valid average MRR: 0.2480180711514224\n",
      "[1]Valid average MR: 170.24288593665946\n",
      "[1]Valid average HITS@1: 0.1765186542010255\n",
      "[1]Valid average HITS@3: 0.2664462708621312\n",
      "[1]Valid average HITS@10: 0.3777279938988268\n",
      "[proc 1]validation take 380.630 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.41300334141105416\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.4127910494655371\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.5707881133899093\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.5707873856768012\n",
      "[proc 0][Train](60000/100000) average loss: 0.49178921754211186\n",
      "[proc 1][Train](60000/100000) average loss: 0.4918957273855805\n",
      "[proc 0][Train](60000/100000) average regularization: 0.024438886102009564\n",
      "[proc 1][Train](60000/100000) average regularization: 0.024438886155281215\n",
      "[proc 0][Train] 20000 steps take 729.954 seconds\n",
      "[proc 0]sample: 64.634, forward: 141.789, backward: 46.801, update: 89.756\n",
      "[proc 1][Train] 20000 steps take 729.952 seconds\n",
      "[proc 1]sample: 65.182, forward: 147.763, backward: 48.756, update: 83.402\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.40942944740653037\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.40920843349397185\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.5610930751517414\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.5611176402017474\n",
      "[proc 0][Train](80000/100000) average loss: 0.4851507543519139\n",
      "[proc 1][Train](80000/100000) average loss: 0.4852735437110066\n",
      "[proc 0][Train](80000/100000) average regularization: 0.025727369085140527\n",
      "[proc 1][Train](80000/100000) average regularization: 0.025730021942779423\n",
      "[proc 0][Train] 20000 steps take 340.490 seconds\n",
      "[proc 0]sample: 63.699, forward: 140.578, backward: 46.871, update: 80.783\n",
      "[proc 1][Train] 20000 steps take 340.488 seconds\n",
      "[proc 1]sample: 63.975, forward: 147.458, backward: 49.002, update: 79.811\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.40683018444925545\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.40701640550643203\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.5522568703606725\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.5522784757018089\n",
      "[proc 0][Train](100000/100000) average loss: 0.47954352741390466\n",
      "[proc 0][Train](100000/100000) average regularization: 0.02684420616608113\n",
      "[proc 1][Train](100000/100000) average loss: 0.4796474406048655\n",
      "[proc 0][Train] 20000 steps take 341.207 seconds\n",
      "[proc 0]sample: 64.479, forward: 143.188, backward: 47.623, update: 82.395\n",
      "[proc 1][Train](100000/100000) average regularization: 0.02684943092931062\n",
      "[proc 1][Train] 20000 steps take 341.209 seconds\n",
      "[proc 1]sample: 63.430, forward: 147.076, backward: 48.723, update: 79.975\n",
      "[0]Valid average MRR: 0.27953768869172535\n",
      "[0]Valid average MR: 144.22441371139075\n",
      "[0]Valid average HITS@1: 0.19776175300975105\n",
      "[0]Valid average HITS@3: 0.3031915618020374\n",
      "[0]Valid average HITS@10: 0.43341096039657895\n",
      "[proc 0]validation take 364.620 seconds:\n",
      "[1]Valid average MRR: 0.27987822388440436\n",
      "[1]Valid average MR: 141.42094350286334\n",
      "[1]Valid average HITS@1: 0.19814513438242645\n",
      "[1]Valid average HITS@3: 0.30309416643401405\n",
      "[1]Valid average HITS@10: 0.4334011998066146\n",
      "[proc 1]validation take 369.695 seconds:\n",
      "proc 1 takes 2490.019 seconds\n",
      "proc 0 takes 2490.019 seconds\n",
      "training takes 2490.5198843479156 seconds\n",
      "Save model to ckpts/DistMult_DRKG_0\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.27914858244200014\n",
      "Test average MR : 144.06824325704596\n",
      "Test average HITS@1 : 0.19753399565563778\n",
      "Test average HITS@3 : 0.30259197722954984\n",
      "Test average HITS@10 : 0.43253641297316436\n",
      "-----------------------------------------\n",
      "testing takes 359.022 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 200 \\\n",
    "--gamma 50.0 --lr 0.01 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: **200**, 400\n",
    "\n",
    "- gamma: **50**, 125, 200\n",
    "\n",
    "- lr: 0.01, **0.05**, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.438 seconds\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.40029544595479966\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.40069517919272185\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.5240841154903173\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.525641973157227\n",
      "[proc 0][Train](20000/100000) average loss: 0.46238964741081\n",
      "[proc 1][Train](20000/100000) average loss: 0.46296870943009855\n",
      "[proc 0][Train](20000/100000) average regularization: 0.0254632799379935\n",
      "[proc 1][Train](20000/100000) average regularization: 0.025405513223598245\n",
      "[proc 0][Train] 20000 steps take 425.216 seconds\n",
      "[proc 0]sample: 68.003, forward: 153.047, backward: 53.345, update: 150.534\n",
      "[proc 1][Train] 20000 steps take 425.218 seconds\n",
      "[proc 1]sample: 65.093, forward: 151.070, backward: 48.037, update: 89.169\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.3808642697930336\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.3806845164522529\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.4912560575336218\n",
      "[proc 1][Train](40000/100000) average loss: 0.43606016360372307\n",
      "[proc 1][Train](40000/100000) average regularization: 0.030123805721569807\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.49128596501201394\n",
      "[proc 1][Train] 20000 steps take 363.208 seconds\n",
      "[proc 1]sample: 64.897, forward: 151.494, backward: 47.943, update: 88.356\n",
      "[proc 0][Train](40000/100000) average loss: 0.4359852407172322\n",
      "[proc 0][Train](40000/100000) average regularization: 0.03011853655865416\n",
      "[proc 0][Train] 20000 steps take 363.224 seconds\n",
      "[proc 0]sample: 67.452, forward: 153.437, backward: 51.347, update: 90.677\n",
      "[0]Valid average MRR: 0.4037113311056567\n",
      "[0]Valid average MR: 77.37076115378329\n",
      "[0]Valid average HITS@1: 0.30611960015253037\n",
      "[0]Valid average HITS@3: 0.44178310726153514\n",
      "[0]Valid average HITS@10: 0.5899827041455575\n",
      "[proc 0]validation take 394.182 seconds:\n",
      "[1]Valid average MRR: 0.4043746122622299\n",
      "[1]Valid average MR: 75.91635400423542\n",
      "[1]Valid average HITS@1: 0.3068052595381902\n",
      "[1]Valid average HITS@3: 0.44247124754012407\n",
      "[1]Valid average HITS@10: 0.5904110801664204\n",
      "[proc 1]validation take 404.565 seconds:\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.3689641921952367\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.36908182585537436\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.48193737075924875\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.4819137024953961\n",
      "[proc 0][Train](60000/100000) average loss: 0.42545078144669535\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03183246441492811\n",
      "[proc 1][Train](60000/100000) average loss: 0.42549776416420937\n",
      "[proc 0][Train] 20000 steps take 760.973 seconds\n",
      "[proc 0]sample: 66.126, forward: 150.788, backward: 50.661, update: 88.545\n",
      "[proc 1][Train](60000/100000) average regularization: 0.031841849267017094\n",
      "[proc 1][Train] 20000 steps take 760.990 seconds\n",
      "[proc 1]sample: 63.884, forward: 150.091, backward: 48.111, update: 87.728\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.3621936322182417\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.3623043543890119\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.47692626102417707\n",
      "[proc 0][Train](80000/100000) average loss: 0.4195599466368556\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.476780731767416\n",
      "[proc 0][Train](80000/100000) average regularization: 0.032816560430079696\n",
      "[proc 1][Train](80000/100000) average loss: 0.4195425431147218\n",
      "[proc 0][Train] 20000 steps take 356.586 seconds\n",
      "[proc 0]sample: 66.649, forward: 150.321, backward: 50.949, update: 88.187\n",
      "[proc 1][Train](80000/100000) average regularization: 0.032826672703772784\n",
      "[proc 1][Train] 20000 steps take 356.587 seconds\n",
      "[proc 1]sample: 64.282, forward: 150.273, backward: 48.226, update: 87.392\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.35773079537302255\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.35761784213483333\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.4735716930910945\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.473450323985517\n",
      "[proc 0][Train](100000/100000) average loss: 0.4155947676435113\n",
      "[proc 1][Train](100000/100000) average loss: 0.4155905597135425\n",
      "[proc 0][Train](100000/100000) average regularization: 0.033506241324357686\n",
      "[proc 1][Train](100000/100000) average regularization: 0.03351845130398869\n",
      "[proc 0][Train] 20000 steps take 362.671 seconds\n",
      "[proc 0]sample: 67.387, forward: 154.103, backward: 51.364, update: 89.523\n",
      "[proc 1][Train] 20000 steps take 362.667 seconds\n",
      "[proc 1]sample: 64.953, forward: 152.432, backward: 48.151, update: 88.316\n",
      "[1]Valid average MRR: 0.42850420681476986\n",
      "[1]Valid average MR: 77.03553456764062\n",
      "[1]Valid average HITS@1: 0.3321768795494937\n",
      "[1]Valid average HITS@3: 0.46704276949685747\n",
      "[1]Valid average HITS@10: 0.611806042612882\n",
      "[proc 1]validation take 382.103 seconds:\n",
      "[0]Valid average MRR: 0.4273559031623114\n",
      "[0]Valid average MR: 78.36405390314322\n",
      "[0]Valid average HITS@1: 0.3306742659475949\n",
      "[0]Valid average HITS@3: 0.466147110094242\n",
      "[0]Valid average HITS@10: 0.6114867625428991\n",
      "[proc 0]validation take 388.221 seconds:\n",
      "proc 0 takes 2656.890 seconds\n",
      "proc 1 takes 2656.890 seconds\n",
      "training takes 2657.387754917145 seconds\n",
      "Save model to ckpts/DistMult_DRKG_1\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.42651081499330645\n",
      "Test average MR : 77.71459310758084\n",
      "Test average HITS@1 : 0.3296335891377326\n",
      "Test average HITS@3 : 0.46495910988240263\n",
      "Test average HITS@10 : 0.6112732113552639\n",
      "-----------------------------------------\n",
      "testing takes 383.558 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 200 \\\n",
    "--gamma 50.0 --lr 0.05 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: **200**, 400\n",
    "\n",
    "- gamma: **50**, 125, 200\n",
    "\n",
    "- lr: 0.01, 0.05, **0.1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.386 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.3697479482918978\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.3705442703962326\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.5007138380482793\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.5011041799411178\n",
      "[proc 0][Train](20000/100000) average loss: 0.43523089328855274\n",
      "[proc 1][Train](20000/100000) average loss: 0.4358242250487208\n",
      "[proc 0][Train](20000/100000) average regularization: 0.028166449290263698\n",
      "[proc 0][Train] 20000 steps take 372.339 seconds\n",
      "[proc 0]sample: 68.729, forward: 159.020, backward: 52.152, update: 92.166\n",
      "[proc 1][Train](20000/100000) average regularization: 0.028196175976714585\n",
      "[proc 1][Train] 20000 steps take 372.336 seconds\n",
      "[proc 1]sample: 66.496, forward: 156.294, backward: 47.958, update: 91.638\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.34727380831837656\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.346732590842247\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.47208735534101726\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.47205890597552064\n",
      "[proc 1][Train](40000/100000) average loss: 0.4096805817872286\n",
      "[proc 1][Train](40000/100000) average regularization: 0.03366616087500006\n",
      "[proc 0][Train](40000/100000) average loss: 0.4093957483559847\n",
      "[proc 1][Train] 20000 steps take 364.706 seconds\n",
      "[proc 1]sample: 66.035, forward: 154.031, backward: 48.280, update: 90.127\n",
      "[proc 0][Train](40000/100000) average regularization: 0.03366935793776065\n",
      "[proc 0][Train] 20000 steps take 364.717 seconds\n",
      "[proc 0]sample: 66.954, forward: 155.546, backward: 51.911, update: 90.022\n",
      "[0]Valid average MRR: 0.4449015579712131\n",
      "[0]Valid average MR: 82.95825502533094\n",
      "[0]Valid average HITS@1: 0.35025126654682137\n",
      "[0]Valid average HITS@3: 0.48292885547747455\n",
      "[0]Valid average HITS@10: 0.6256877485427902\n",
      "[proc 0]validation take 337.734 seconds:\n",
      "[1]Valid average MRR: 0.4435640566689997\n",
      "[1]Valid average MR: 81.32420994572952\n",
      "[1]Valid average HITS@1: 0.3489585106600298\n",
      "[1]Valid average HITS@3: 0.4810291644252572\n",
      "[1]Valid average HITS@10: 0.6245497320522685\n",
      "[proc 1]validation take 354.130 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.3421205643504858\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.34166968421936034\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.46711301375478503\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.4670375036329031\n",
      "[proc 1][Train](60000/100000) average loss: 0.4046167890533805\n",
      "[proc 0][Train](60000/100000) average loss: 0.404353593967855\n",
      "[proc 1][Train](60000/100000) average regularization: 0.0352138881534338\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03521735223717987\n",
      "[proc 1][Train] 20000 steps take 718.381 seconds\n",
      "[proc 1]sample: 65.578, forward: 151.944, backward: 48.366, update: 89.299\n",
      "[proc 0][Train] 20000 steps take 718.375 seconds\n",
      "[proc 0]sample: 66.046, forward: 156.191, backward: 51.721, update: 90.011\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.33913708715885876\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.33952521324157714\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.4644025175571442\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.4646554313376546\n",
      "[proc 0][Train](80000/100000) average loss: 0.4017698023915291\n",
      "[proc 1][Train](80000/100000) average loss: 0.4020903223350644\n",
      "[proc 0][Train](80000/100000) average regularization: 0.03612458087913692\n",
      "[proc 1][Train](80000/100000) average regularization: 0.036121389661543074\n",
      "[proc 0][Train] 20000 steps take 370.248 seconds\n",
      "[proc 0]sample: 66.117, forward: 159.098, backward: 51.708, update: 93.057\n",
      "[proc 1][Train] 20000 steps take 370.251 seconds\n",
      "[proc 1]sample: 65.948, forward: 154.653, backward: 48.598, update: 88.532\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.33783279555141926\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.3374533745095134\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.4628493528738618\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.4628126411020756\n",
      "[proc 1][Train](100000/100000) average loss: 0.4003410742267966\n",
      "[proc 0][Train](100000/100000) average loss: 0.40013300773352384\n",
      "[proc 1][Train](100000/100000) average regularization: 0.036749647498317065\n",
      "[proc 1][Train] 20000 steps take 374.157 seconds\n",
      "[proc 1]sample: 65.758, forward: 153.130, backward: 48.346, update: 88.851\n",
      "[proc 0][Train](100000/100000) average regularization: 0.03675210780277848\n",
      "[proc 0][Train] 20000 steps take 374.162 seconds\n",
      "[proc 0]sample: 66.953, forward: 159.298, backward: 52.093, update: 95.537\n",
      "[0]Valid average MRR: 0.45503971916323466\n",
      "[0]Valid average MR: 88.02195347823718\n",
      "[0]Valid average HITS@1: 0.3623481505692651\n",
      "[0]Valid average HITS@3: 0.49167551887563327\n",
      "[0]Valid average HITS@10: 0.6325311870131285\n",
      "[proc 0]validation take 333.890 seconds:\n",
      "[1]Valid average MRR: 0.45545291082901107\n",
      "[1]Valid average MR: 86.00564494712543\n",
      "[1]Valid average HITS@1: 0.3630947111816257\n",
      "[1]Valid average HITS@3: 0.49154960267471076\n",
      "[1]Valid average HITS@10: 0.6321659845972613\n",
      "[proc 1]validation take 344.331 seconds:\n",
      "proc 1 takes 2544.164 seconds\n",
      "proc 0 takes 2544.169 seconds\n",
      "training takes 2544.665165424347 seconds\n",
      "Save model to ckpts/DistMult_DRKG_2\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.4535011863034895\n",
      "Test average MR : 87.93707824618507\n",
      "Test average HITS@1 : 0.3604680062918349\n",
      "Test average HITS@3 : 0.49067119715097\n",
      "Test average HITS@10 : 0.6309794562056967\n",
      "-----------------------------------------\n",
      "testing takes 353.222 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 200 \\\n",
    "--gamma 50.0 --lr 0.1 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: **200**, 400\n",
    "\n",
    "- gamma: 50, **125**, 200\n",
    "\n",
    "- lr: **0.01**, 0.05, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.263 seconds\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.4636830395489931\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.4631375725805759\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.6498785514950752\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.6498611712157726\n",
      "[proc 1][Train](20000/100000) average loss: 0.5567807957112789\n",
      "[proc 1][Train](20000/100000) average regularization: 0.023433260954963044\n",
      "[proc 0][Train](20000/100000) average loss: 0.5564993717148901\n",
      "[proc 1][Train] 20000 steps take 378.413 seconds\n",
      "[proc 1]sample: 64.727, forward: 151.948, backward: 46.615, update: 87.780\n",
      "[proc 0][Train](20000/100000) average regularization: 0.023451842741155998\n",
      "[proc 0][Train] 20000 steps take 378.430 seconds\n",
      "[proc 0]sample: 67.528, forward: 153.130, backward: 50.998, update: 106.462\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.440817885415256\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.4403912317618728\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.620500344645977\n",
      "[proc 1][Train](40000/100000) average loss: 0.5306591152071953\n",
      "[proc 1][Train](40000/100000) average regularization: 0.027252764382679016\n",
      "[proc 1][Train] 20000 steps take 404.040 seconds\n",
      "[proc 1]sample: 64.324, forward: 146.780, backward: 46.684, update: 84.863\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.6205781968355178\n",
      "[proc 0][Train](40000/100000) average loss: 0.5304847141638398\n",
      "[proc 0][Train](40000/100000) average regularization: 0.02725772393476218\n",
      "[proc 0][Train] 20000 steps take 404.047 seconds\n",
      "[proc 0]sample: 67.161, forward: 148.473, backward: 52.435, update: 135.658\n",
      "[0]Valid average MRR: 0.18807123484932847\n",
      "[0]Valid average MR: 304.2612661382579\n",
      "[0]Valid average HITS@1: 0.12831276897096475\n",
      "[0]Valid average HITS@3: 0.20336928692052078\n",
      "[0]Valid average HITS@10: 0.29210246772348425\n",
      "[proc 0]validation take 340.055 seconds:\n",
      "[1]Valid average MRR: 0.18823691184913463\n",
      "[1]Valid average MR: 298.79963161442765\n",
      "[1]Valid average HITS@1: 0.12879876342292162\n",
      "[1]Valid average HITS@3: 0.20311255166590628\n",
      "[1]Valid average HITS@10: 0.29136506942127377\n",
      "[proc 1]validation take 361.362 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.43513050454258917\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.4346612539365888\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.6173357244759798\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.6172600416362286\n",
      "[proc 1][Train](60000/100000) average loss: 0.5262331144884229\n",
      "[proc 1][Train](60000/100000) average regularization: 0.02788734916979447\n",
      "[proc 0][Train](60000/100000) average loss: 0.5259606478348374\n",
      "[proc 1][Train] 20000 steps take 782.081 seconds\n",
      "[proc 1]sample: 63.346, forward: 144.297, backward: 46.947, update: 90.716\n",
      "[proc 0][Train](60000/100000) average regularization: 0.027892571916617453\n",
      "[proc 0][Train] 20000 steps take 782.078 seconds\n",
      "[proc 0]sample: 66.544, forward: 147.660, backward: 51.613, update: 154.581\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.42968008226901294\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.4301997970879078\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.6145460918724537\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.6144099467277527\n",
      "[proc 0][Train](80000/100000) average loss: 0.5221130869582296\n",
      "[proc 1][Train](80000/100000) average loss: 0.5223048719719052\n",
      "[proc 0][Train](80000/100000) average regularization: 0.028416046385560185\n",
      "[proc 0][Train] 20000 steps take 390.883 seconds\n",
      "[proc 0]sample: 65.258, forward: 146.848, backward: 50.292, update: 128.187\n",
      "[proc 1][Train](80000/100000) average regularization: 0.028411691028345377\n",
      "[proc 1][Train] 20000 steps take 390.897 seconds\n",
      "[proc 1]sample: 63.552, forward: 145.345, backward: 46.835, update: 84.390\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.42554873800873755\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.6112782570451498\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.42606762380450963\n",
      "[proc 0][Train](100000/100000) average loss: 0.5184134975656867\n",
      "[proc 0][Train](100000/100000) average regularization: 0.028841489040944725\n",
      "[proc 0][Train] 20000 steps take 345.636 seconds\n",
      "[proc 0]sample: 64.028, forward: 145.615, backward: 49.424, update: 86.127\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.6112601107865572\n",
      "[proc 1][Train](100000/100000) average loss: 0.5186638674035668\n",
      "[proc 1][Train](100000/100000) average regularization: 0.028840710722561926\n",
      "[proc 1][Train] 20000 steps take 345.652 seconds\n",
      "[proc 1]sample: 63.347, forward: 146.767, backward: 46.838, update: 85.126\n",
      "[0]Valid average MRR: 0.21115184783331126\n",
      "[0]Valid average MR: 257.94436727134064\n",
      "[0]Valid average HITS@1: 0.14700795336928693\n",
      "[0]Valid average HITS@3: 0.22849934629841478\n",
      "[0]Valid average HITS@10: 0.3246445497630332\n",
      "[proc 0]validation take 319.712 seconds:\n",
      "[1]Valid average MRR: 0.21150550746475802\n",
      "[1]Valid average MR: 253.28818170056584\n",
      "[1]Valid average HITS@1: 0.14707164112027346\n",
      "[1]Valid average HITS@3: 0.2295396201747278\n",
      "[1]Valid average HITS@10: 0.32500663911151667\n",
      "[proc 1]validation take 345.071 seconds:\n",
      "proc 1 takes 2646.154 seconds\n",
      "proc 0 takes 2646.164 seconds\n",
      "training takes 2646.6341445446014 seconds\n",
      "Save model to ckpts/DistMult_DRKG_3\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.20985014406044314\n",
      "Test average MR : 256.3819974533049\n",
      "Test average HITS@1 : 0.14548676603771016\n",
      "Test average HITS@3 : 0.22715124236502177\n",
      "Test average HITS@10 : 0.32313577153285167\n",
      "-----------------------------------------\n",
      "testing takes 339.832 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 200 \\\n",
    "--gamma 125.0 --lr 0.01 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: **200**, 400\n",
    "\n",
    "- gamma: 50, **125**, 200\n",
    "\n",
    "- lr: 0.01, **0.05**, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.298 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.40686404414772986\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.4068410370916128\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.5964312821388245\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.5967325039222836\n",
      "[proc 0][Train](20000/100000) average loss: 0.5016476630613208\n",
      "[proc 1][Train](20000/100000) average loss: 0.5017867705523967\n",
      "[proc 0][Train](20000/100000) average regularization: 0.030246382289892064\n",
      "[proc 0][Train] 20000 steps take 364.646 seconds\n",
      "[proc 0]sample: 68.648, forward: 154.113, backward: 50.609, update: 90.963\n",
      "[proc 1][Train](20000/100000) average regularization: 0.030233624513773247\n",
      "[proc 1][Train] 20000 steps take 364.649 seconds\n",
      "[proc 1]sample: 65.812, forward: 150.922, backward: 46.497, update: 87.603\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.3913907345443964\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.39140994117558003\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.5267738691031932\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.5264976063489913\n",
      "[proc 1][Train](40000/100000) average loss: 0.45908230178058146\n",
      "[proc 0][Train](40000/100000) average loss: 0.4589537737995386\n",
      "[proc 1][Train](40000/100000) average regularization: 0.03488148237094283\n",
      "[proc 0][Train](40000/100000) average regularization: 0.034888285274617375\n",
      "[proc 1][Train] 20000 steps take 357.353 seconds\n",
      "[proc 1]sample: 65.363, forward: 147.220, backward: 46.501, update: 85.946\n",
      "[proc 0][Train] 20000 steps take 357.356 seconds\n",
      "[proc 0]sample: 67.646, forward: 149.988, backward: 50.544, update: 88.862\n",
      "[1]Valid average MRR: 0.3470516750269407\n",
      "[1]Valid average MR: 102.72136159665526\n",
      "[1]Valid average HITS@1: 0.2542030682909225\n",
      "[1]Valid average HITS@3: 0.37855532933397795\n",
      "[1]Valid average HITS@10: 0.5266449675534703\n",
      "[proc 1]validation take 387.647 seconds:\n",
      "[0]Valid average MRR: 0.34752689389920727\n",
      "[0]Valid average MR: 102.49117502859944\n",
      "[0]Valid average HITS@1: 0.2547801928419676\n",
      "[0]Valid average HITS@3: 0.37999128397886367\n",
      "[0]Valid average HITS@10: 0.525385411559623\n",
      "[proc 0]validation take 407.697 seconds:\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.38928618098944423\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.38921761113107206\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.5002832793936134\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.5004656095132232\n",
      "[proc 0][Train](60000/100000) average loss: 0.44478473014831543\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03453546491209417\n",
      "[proc 1][Train](60000/100000) average loss: 0.44484161027520897\n",
      "[proc 0][Train] 20000 steps take 767.996 seconds\n",
      "[proc 0]sample: 66.524, forward: 148.493, backward: 50.012, update: 88.891\n",
      "[proc 1][Train](60000/100000) average regularization: 0.034529419948346914\n",
      "[proc 1][Train] 20000 steps take 768.005 seconds\n",
      "[proc 1]sample: 64.611, forward: 147.868, backward: 47.230, update: 93.112\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.38304578951746227\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.38307625512182714\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.4930683009251952\n",
      "[proc 1][Train](80000/100000) average loss: 0.43805704518556593\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.4931341527238488\n",
      "[proc 1][Train](80000/100000) average regularization: 0.03472835235614329\n",
      "[proc 0][Train](80000/100000) average loss: 0.4381052040576935\n",
      "[proc 1][Train] 20000 steps take 382.170 seconds\n",
      "[proc 1]sample: 64.845, forward: 143.233, backward: 48.424, update: 125.383\n",
      "[proc 0][Train](80000/100000) average regularization: 0.034733793366327884\n",
      "[proc 0][Train] 20000 steps take 382.183 seconds\n",
      "[proc 0]sample: 67.263, forward: 146.286, backward: 49.862, update: 87.008\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.377381754770875\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.3772677569568157\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.48798703260570764\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.488138475266099\n",
      "[proc 1][Train](100000/100000) average loss: 0.4326273947402835\n",
      "[proc 0][Train](100000/100000) average loss: 0.43276011503636835\n",
      "[proc 1][Train](100000/100000) average regularization: 0.03514728937167674\n",
      "[proc 0][Train](100000/100000) average regularization: 0.035154112999141215\n",
      "[proc 1][Train] 20000 steps take 391.250 seconds\n",
      "[proc 1]sample: 64.935, forward: 143.975, backward: 48.480, update: 126.455\n",
      "[proc 0][Train] 20000 steps take 391.247 seconds\n",
      "[proc 0]sample: 68.011, forward: 148.779, backward: 51.630, update: 112.991\n",
      "[1]Valid average MRR: 0.3887170930490174\n",
      "[1]Valid average MR: 95.09525933390985\n",
      "[1]Valid average HITS@1: 0.2937074841512492\n",
      "[1]Valid average HITS@3: 0.4247669501624029\n",
      "[1]Valid average HITS@10: 0.5707286680239961\n",
      "[proc 1]validation take 384.847 seconds:\n",
      "[0]Valid average MRR: 0.3884288981310466\n",
      "[0]Valid average MR: 94.89923121969821\n",
      "[0]Valid average HITS@1: 0.2935698915944871\n",
      "[0]Valid average HITS@3: 0.4245791796045105\n",
      "[0]Valid average HITS@10: 0.570150351364602\n",
      "[proc 0]validation take 408.939 seconds:\n",
      "proc 0 takes 2672.369 seconds\n",
      "proc 1 takes 2672.368 seconds\n",
      "training takes 2672.857306241989 seconds\n",
      "Save model to ckpts/DistMult_DRKG_4\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.3876524888715606\n",
      "Test average MR : 96.19122343504225\n",
      "Test average HITS@1 : 0.2930214426278625\n",
      "Test average HITS@3 : 0.42346296056708227\n",
      "Test average HITS@10 : 0.5687777906398742\n",
      "-----------------------------------------\n",
      "testing takes 386.541 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 200 \\\n",
    "--gamma 125.0 --lr 0.05 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: **200**, 400\n",
    "\n",
    "- gamma: 50, **125**, 200\n",
    "\n",
    "- lr: 0.01, 0.05, **0.1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.173 seconds\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.39375939021855594\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.39479281584471465\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.5217447260841728\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.5194535019516945\n",
      "[proc 1][Train](20000/100000) average loss: 0.4577520582139492\n",
      "[proc 0][Train](20000/100000) average loss: 0.4571231588438153\n",
      "[proc 1][Train](20000/100000) average regularization: 0.02917581582288258\n",
      "[proc 1][Train] 20000 steps take 421.509 seconds\n",
      "[proc 1]sample: 65.513, forward: 149.961, backward: 47.434, update: 88.523\n",
      "[proc 0][Train](20000/100000) average regularization: 0.029244746448937802\n",
      "[proc 0][Train] 20000 steps take 421.516 seconds\n",
      "[proc 0]sample: 67.774, forward: 155.967, backward: 52.992, update: 144.485\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.36317502245604993\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.36303678405582906\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.48314287458211186\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.4829982199713588\n",
      "[proc 0][Train](40000/100000) average loss: 0.4231589486345649\n",
      "[proc 1][Train](40000/100000) average loss: 0.4230175019934773\n",
      "[proc 0][Train](40000/100000) average regularization: 0.034486016080155966\n",
      "[proc 0][Train] 20000 steps take 364.243 seconds\n",
      "[proc 0]sample: 66.968, forward: 155.754, backward: 50.912, update: 90.310\n",
      "[proc 1][Train](40000/100000) average regularization: 0.034485624392516914\n",
      "[proc 1][Train] 20000 steps take 364.252 seconds\n",
      "[proc 1]sample: 65.833, forward: 149.263, backward: 46.925, update: 88.340\n",
      "[1]Valid average MRR: 0.42319887778699083\n",
      "[1]Valid average MR: 89.83482911948357\n",
      "[1]Valid average HITS@1: 0.3285951640030778\n",
      "[1]Valid average HITS@3: 0.4605228215202544\n",
      "[1]Valid average HITS@10: 0.6033386219247295\n",
      "[proc 1]validation take 376.288 seconds:\n",
      "[0]Valid average MRR: 0.42348423292102116\n",
      "[0]Valid average MR: 91.91395993354034\n",
      "[0]Valid average HITS@1: 0.328468023097456\n",
      "[0]Valid average HITS@3: 0.46138394073105626\n",
      "[0]Valid average HITS@10: 0.6037342703056054\n",
      "[proc 0]validation take 382.556 seconds:\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.3536671718418598\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.35356173869073393\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.47470825590491295\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.47477466255277395\n",
      "[proc 0][Train](60000/100000) average loss: 0.4141877138718963\n",
      "[proc 1][Train](60000/100000) average loss: 0.41416820069253446\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03587595671508461\n",
      "[proc 1][Train](60000/100000) average regularization: 0.03587870469577611\n",
      "[proc 0][Train] 20000 steps take 745.973 seconds\n",
      "[proc 0]sample: 66.395, forward: 155.690, backward: 50.688, update: 90.353\n",
      "[proc 1][Train] 20000 steps take 745.970 seconds\n",
      "[proc 1]sample: 65.047, forward: 150.764, backward: 47.218, update: 87.778\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.34891639274060726\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.3487843170151114\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.47054187661260366\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.47062032591253516\n",
      "[proc 0][Train](80000/100000) average loss: 0.4097291346341371\n",
      "[proc 1][Train](80000/100000) average loss: 0.4097023214906454\n",
      "[proc 0][Train](80000/100000) average regularization: 0.036613266528211535\n",
      "[proc 0][Train] 20000 steps take 357.539 seconds\n",
      "[proc 0]sample: 65.153, forward: 152.092, backward: 50.359, update: 89.544\n",
      "[proc 1][Train](80000/100000) average regularization: 0.03661434774696827\n",
      "[proc 1][Train] 20000 steps take 357.540 seconds\n",
      "[proc 1]sample: 65.189, forward: 149.905, backward: 47.332, update: 87.610\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.3456705556452274\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.46770728578567505\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.34557353321164846\n",
      "[proc 0][Train](100000/100000) average loss: 0.40668892067372797\n",
      "[proc 0][Train](100000/100000) average regularization: 0.03713462142795324\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.4677606689557433\n",
      "[proc 0][Train] 20000 steps take 358.631 seconds\n",
      "[proc 0]sample: 64.891, forward: 153.512, backward: 50.618, update: 89.322\n",
      "[proc 1][Train](100000/100000) average loss: 0.4066671010196209\n",
      "[proc 1][Train](100000/100000) average regularization: 0.03713707255404443\n",
      "[proc 1][Train] 20000 steps take 358.640 seconds\n",
      "[proc 1]sample: 64.788, forward: 150.657, backward: 47.567, update: 87.691\n",
      "[0]Valid average MRR: 0.4421876367310942\n",
      "[0]Valid average MR: 94.19057444026802\n",
      "[0]Valid average HITS@1: 0.34832080405294985\n",
      "[0]Valid average HITS@3: 0.47969779920466304\n",
      "[0]Valid average HITS@10: 0.6211867407528463\n",
      "[proc 0]validation take 374.467 seconds:\n",
      "[1]Valid average MRR: 0.441997598957335\n",
      "[1]Valid average MR: 92.46218089706313\n",
      "[1]Valid average HITS@1: 0.3484444051015614\n",
      "[1]Valid average HITS@3: 0.47903402629769093\n",
      "[1]Valid average HITS@10: 0.6208999230543999\n",
      "[proc 1]validation take 384.821 seconds:\n",
      "proc 1 takes 2632.732 seconds\n",
      "proc 0 takes 2632.735 seconds\n",
      "training takes 2633.2217922210693 seconds\n",
      "Save model to ckpts/DistMult_DRKG_5\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.4409446284074526\n",
      "Test average MR : 95.14563997630349\n",
      "Test average HITS@1 : 0.3469752888864678\n",
      "Test average HITS@3 : 0.4785914188632479\n",
      "Test average HITS@10 : 0.6200436479023812\n",
      "-----------------------------------------\n",
      "testing takes 390.753 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 200 \\\n",
    "--gamma 125.0 --lr 0.1 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: **200**, 400\n",
    "\n",
    "- gamma: 50, 125, **200**\n",
    "\n",
    "- lr: **0.01**, 0.05, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.299 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.5182063047111034\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.5181976607844233\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.7419600798189641\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.7419044578105212\n",
      "[proc 0][Train](20000/100000) average loss: 0.6300831921756268\n",
      "[proc 0][Train](20000/100000) average regularization: 0.039160269008763134\n",
      "[proc 1][Train](20000/100000) average loss: 0.6300510591447354\n",
      "[proc 0][Train] 20000 steps take 373.205 seconds\n",
      "[proc 0]sample: 68.089, forward: 159.606, backward: 50.808, update: 94.344\n",
      "[proc 1][Train](20000/100000) average regularization: 0.039155634704604744\n",
      "[proc 1][Train] 20000 steps take 373.193 seconds\n",
      "[proc 1]sample: 68.109, forward: 157.118, backward: 49.277, update: 91.482\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.48831843982189893\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.4882061515972018\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.6603511078089476\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.6602450694054365\n",
      "[proc 0][Train](40000/100000) average loss: 0.5743347739756107\n",
      "[proc 1][Train](40000/100000) average loss: 0.5742256103813648\n",
      "[proc 0][Train](40000/100000) average regularization: 0.04194131751377136\n",
      "[proc 1][Train](40000/100000) average regularization: 0.04194057370573282\n",
      "[proc 0][Train] 20000 steps take 369.529 seconds\n",
      "[proc 0]sample: 68.173, forward: 157.051, backward: 51.512, update: 92.522\n",
      "[proc 1][Train] 20000 steps take 369.525 seconds\n",
      "[proc 1]sample: 67.365, forward: 153.288, backward: 47.965, update: 90.431\n",
      "[1]Valid average MRR: 0.08429151131177277\n",
      "[1]Valid average MR: 420.9807091252034\n",
      "[1]Valid average HITS@1: 0.0467631777851924\n",
      "[1]Valid average HITS@3: 0.08490572461646363\n",
      "[1]Valid average HITS@10: 0.14860374377796087\n",
      "[proc 1]validation take 393.711 seconds:\n",
      "[0]Valid average MRR: 0.08373167957353711\n",
      "[0]Valid average MR: 426.5809466416081\n",
      "[0]Valid average HITS@1: 0.04614384158631585\n",
      "[0]Valid average HITS@3: 0.08463392711227324\n",
      "[0]Valid average HITS@10: 0.14905417551887562\n",
      "[proc 0]validation take 395.563 seconds:\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.49521770900636913\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.4951371171504259\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.6453657909631729\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.6454844244420529\n",
      "[proc 0][Train](60000/100000) average loss: 0.5702917501032353\n",
      "[proc 1][Train](60000/100000) average loss: 0.570310770753026\n",
      "[proc 0][Train](60000/100000) average regularization: 0.04113544442765415\n",
      "[proc 1][Train](60000/100000) average regularization: 0.04113415725100786\n",
      "[proc 0][Train] 20000 steps take 766.297 seconds\n",
      "[proc 0]sample: 67.330, forward: 158.112, backward: 51.617, update: 93.402\n",
      "[proc 1][Train] 20000 steps take 766.299 seconds\n",
      "[proc 1]sample: 66.646, forward: 155.556, backward: 48.095, update: 90.319\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.4873429482549429\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.48741631880700587\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.6380428743243217\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.6381569125115871\n",
      "[proc 1][Train](80000/100000) average loss: 0.5626929112046957\n",
      "[proc 0][Train](80000/100000) average loss: 0.5627866158992052\n",
      "[proc 1][Train](80000/100000) average regularization: 0.04086671014074236\n",
      "[proc 0][Train](80000/100000) average regularization: 0.040866204171441496\n",
      "[proc 1][Train] 20000 steps take 371.415 seconds\n",
      "[proc 1]sample: 67.649, forward: 155.536, backward: 48.201, update: 90.477\n",
      "[proc 0][Train] 20000 steps take 371.418 seconds\n",
      "[proc 0]sample: 68.464, forward: 157.897, backward: 51.914, update: 92.867\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.4805316574513912\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.4804974835380912\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.6339917549550533\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.6339002425938844\n",
      "[proc 0][Train](100000/100000) average loss: 0.5572617062658072\n",
      "[proc 1][Train](100000/100000) average loss: 0.55719886302948\n",
      "[proc 0][Train](100000/100000) average regularization: 0.04063649360910058\n",
      "[proc 1][Train](100000/100000) average regularization: 0.0406372523246333\n",
      "[proc 0][Train] 20000 steps take 371.921 seconds\n",
      "[proc 0]sample: 68.014, forward: 158.525, backward: 51.548, update: 93.562\n",
      "[proc 1][Train] 20000 steps take 371.923 seconds\n",
      "[proc 1]sample: 67.003, forward: 154.564, backward: 48.142, update: 90.401\n",
      "[1]Valid average MRR: 0.1293204492557603\n",
      "[1]Valid average MR: 371.36844004712066\n",
      "[1]Valid average HITS@1: 0.08003023349244504\n",
      "[1]Valid average HITS@3: 0.1396051941684768\n",
      "[1]Valid average HITS@10: 0.21442287395221202\n",
      "[proc 1]validation take 380.834 seconds:\n",
      "[0]Valid average MRR: 0.1292708182304253\n",
      "[0]Valid average MR: 375.4471795500354\n",
      "[0]Valid average HITS@1: 0.08008184888598355\n",
      "[0]Valid average HITS@3: 0.1395891213161192\n",
      "[0]Valid average HITS@10: 0.21357316010241326\n",
      "[proc 0]validation take 391.624 seconds:\n",
      "proc 0 takes 2643.994 seconds\n",
      "proc 1 takes 2643.977 seconds\n",
      "training takes 2644.512831926346 seconds\n",
      "Save model to ckpts/DistMult_DRKG_6\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.1298025958176355\n",
      "Test average MR : 375.33470995594354\n",
      "Test average HITS@1 : 0.08102780255622817\n",
      "Test average HITS@3 : 0.1397703207882498\n",
      "Test average HITS@10 : 0.21306951660458814\n",
      "-----------------------------------------\n",
      "testing takes 389.604 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 200 \\\n",
    "--gamma 200.0 --lr 0.01 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: **200**, 400\n",
    "\n",
    "- gamma: 50, 125, **200**\n",
    "\n",
    "- lr: 0.01, **0.05**, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.385 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.4436071756899357\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.44374067816734314\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.6223357828766107\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.6227474636226893\n",
      "[proc 1][Train](20000/100000) average loss: 0.5330382306039333\n",
      "[proc 0][Train](20000/100000) average loss: 0.5331773195847869\n",
      "[proc 1][Train](20000/100000) average regularization: 0.038742702197469774\n",
      "[proc 0][Train](20000/100000) average regularization: 0.03875473017254844\n",
      "[proc 1][Train] 20000 steps take 360.922 seconds\n",
      "[proc 1]sample: 64.849, forward: 150.290, backward: 47.645, update: 93.958\n",
      "[proc 0][Train] 20000 steps take 360.932 seconds\n",
      "[proc 0]sample: 66.828, forward: 154.649, backward: 50.954, update: 88.234\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.45065872030556203\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.4506077608332038\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.5668644675970077\n",
      "[proc 0][Train](40000/100000) average loss: 0.5087615938276052\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.5668159570246935\n",
      "[proc 0][Train](40000/100000) average regularization: 0.03828799068592489\n",
      "[proc 1][Train](40000/100000) average loss: 0.508711858920753\n",
      "[proc 0][Train] 20000 steps take 369.059 seconds\n",
      "[proc 0]sample: 66.826, forward: 152.567, backward: 50.519, update: 98.675\n",
      "[proc 1][Train](40000/100000) average regularization: 0.038290026301890615\n",
      "[proc 1][Train] 20000 steps take 369.066 seconds\n",
      "[proc 1]sample: 65.438, forward: 150.210, backward: 47.413, update: 92.726\n",
      "[0]Valid average MRR: 0.26998624327459025\n",
      "[0]Valid average MR: 140.39686836084329\n",
      "[0]Valid average HITS@1: 0.18783366018412595\n",
      "[0]Valid average HITS@3: 0.2923510105137005\n",
      "[0]Valid average HITS@10: 0.4250694557934303\n",
      "[proc 0]validation take 400.808 seconds:\n",
      "[1]Valid average MRR: 0.27050735696454775\n",
      "[1]Valid average MR: 139.88256262895197\n",
      "[1]Valid average HITS@1: 0.18837712877152604\n",
      "[1]Valid average HITS@3: 0.29269289172460283\n",
      "[1]Valid average HITS@10: 0.4254206472963495\n",
      "[proc 1]validation take 428.071 seconds:\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.4444936511725187\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.44435288546085355\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.5423874171257019\n",
      "[proc 0][Train](60000/100000) average loss: 0.49344053409397604\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.5425215442851186\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03626845911294222\n",
      "[proc 0][Train] 20000 steps take 804.239 seconds\n",
      "[proc 0]sample: 66.371, forward: 148.469, backward: 50.673, update: 110.392\n",
      "[proc 1][Train](60000/100000) average loss: 0.4934372149720788\n",
      "[proc 1][Train](60000/100000) average regularization: 0.03626867237724364\n",
      "[proc 1][Train] 20000 steps take 804.240 seconds\n",
      "[proc 1]sample: 64.877, forward: 147.323, backward: 47.646, update: 91.298\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.4299581201970577\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.43005962095707656\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.530206091670692\n",
      "[proc 1][Train](80000/100000) average loss: 0.4800821060553193\n",
      "[proc 1][Train](80000/100000) average regularization: 0.036024104658886794\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.5302642162278295\n",
      "[proc 1][Train] 20000 steps take 355.451 seconds\n",
      "[proc 1]sample: 64.718, forward: 148.381, backward: 47.553, update: 86.030\n",
      "[proc 0][Train](80000/100000) average loss: 0.4801619186654687\n",
      "[proc 0][Train](80000/100000) average regularization: 0.036026377030834555\n",
      "[proc 0][Train] 20000 steps take 355.469 seconds\n",
      "[proc 0]sample: 67.002, forward: 149.971, backward: 51.013, update: 87.212\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.41751056745648385\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.4174436388909817\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.5200980946093797\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.5202914007619023\n",
      "[proc 0][Train](100000/100000) average loss: 0.46880433104634284\n",
      "[proc 0][Train](100000/100000) average regularization: 0.03631545378025621\n",
      "[proc 1][Train](100000/100000) average loss: 0.4688675198376179\n",
      "[proc 0][Train] 20000 steps take 357.121 seconds\n",
      "[proc 0]sample: 66.476, forward: 151.925, backward: 50.769, update: 87.609\n",
      "[proc 1][Train](100000/100000) average regularization: 0.03631381596624851\n",
      "[proc 1][Train] 20000 steps take 357.136 seconds\n",
      "[proc 1]sample: 64.434, forward: 149.251, backward: 47.860, update: 86.807\n",
      "[0]Valid average MRR: 0.3222050036607892\n",
      "[0]Valid average MR: 115.43503159557662\n",
      "[0]Valid average HITS@1: 0.23162485700277824\n",
      "[0]Valid average HITS@3: 0.3531554720270197\n",
      "[0]Valid average HITS@10: 0.4937898349403497\n",
      "[proc 0]validation take 386.409 seconds:\n",
      "[1]Valid average MRR: 0.3211422850107671\n",
      "[1]Valid average MR: 115.53231715205949\n",
      "[1]Valid average HITS@1: 0.23070061352199758\n",
      "[1]Valid average HITS@3: 0.35091279271672443\n",
      "[1]Valid average HITS@10: 0.49294211375692\n",
      "[proc 1]validation take 407.194 seconds:\n",
      "proc 1 takes 2654.010 seconds\n",
      "proc 0 takes 2654.018 seconds\n",
      "training takes 2654.52792429924 seconds\n",
      "Save model to ckpts/DistMult_DRKG_7\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.3210604764196341\n",
      "Test average MR : 115.61660152393145\n",
      "Test average HITS@1 : 0.2303856813090285\n",
      "Test average HITS@3 : 0.35125836698284724\n",
      "Test average HITS@10 : 0.4931157520581246\n",
      "-----------------------------------------\n",
      "testing takes 387.970 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 200 \\\n",
    "--gamma 200.0 --lr 0.05 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: **200**, 400\n",
    "\n",
    "- gamma: 50, 125, **200**\n",
    "\n",
    "- lr: 0.01, 0.05, **0.1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.140 seconds\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.42567392603605986\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.42609611642956735\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.5435431803569197\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.543165649817884\n",
      "[proc 1][Train](20000/100000) average loss: 0.48460855323672297\n",
      "[proc 0][Train](20000/100000) average loss: 0.4846308830395341\n",
      "[proc 1][Train](20000/100000) average regularization: 0.032344718007743356\n",
      "[proc 0][Train](20000/100000) average regularization: 0.03235437264582142\n",
      "[proc 1][Train] 20000 steps take 366.254 seconds\n",
      "[proc 1]sample: 68.019, forward: 156.580, backward: 49.900, update: 90.862\n",
      "[proc 0][Train] 20000 steps take 366.256 seconds\n",
      "[proc 0]sample: 65.777, forward: 158.046, backward: 47.691, update: 85.914\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.38626704344302415\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.38638600094765424\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.5006016202017665\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.500692536431551\n",
      "[proc 1][Train](40000/100000) average loss: 0.4434343317359686\n",
      "[proc 0][Train](40000/100000) average loss: 0.44353926859349013\n",
      "[proc 1][Train](40000/100000) average regularization: 0.03494151766076684\n",
      "[proc 1][Train] 20000 steps take 364.047 seconds\n",
      "[proc 1]sample: 67.977, forward: 153.204, backward: 50.674, update: 91.908\n",
      "[proc 0][Train](40000/100000) average regularization: 0.03494848932493478\n",
      "[proc 0][Train] 20000 steps take 364.048 seconds\n",
      "[proc 0]sample: 65.048, forward: 153.619, backward: 47.491, update: 82.053\n",
      "[0]Valid average MRR: 0.39186241673463756\n",
      "[0]Valid average MR: 99.37121738301465\n",
      "[0]Valid average HITS@1: 0.2972742005774364\n",
      "[0]Valid average HITS@3: 0.42797366127362857\n",
      "[0]Valid average HITS@10: 0.5722919049953696\n",
      "[proc 0]validation take 398.679 seconds:\n",
      "[1]Valid average MRR: 0.39211163646876546\n",
      "[1]Valid average MR: 98.15782700177724\n",
      "[1]Valid average HITS@1: 0.2981301538231068\n",
      "[1]Valid average HITS@3: 0.427310240574164\n",
      "[1]Valid average HITS@10: 0.5720258482741715\n",
      "[proc 1]validation take 413.648 seconds:\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.3713372283563018\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.37107611026018855\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.48750763363838195\n",
      "[proc 0][Train](60000/100000) average loss: 0.42942243094146254\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.4876630186751485\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03640120671633631\n",
      "[proc 1][Train](60000/100000) average loss: 0.42936956448704006\n",
      "[proc 0][Train] 20000 steps take 826.334 seconds\n",
      "[proc 0]sample: 66.493, forward: 150.537, backward: 49.964, update: 133.081\n",
      "[proc 1][Train](60000/100000) average regularization: 0.036391594833880665\n",
      "[proc 1][Train] 20000 steps take 826.341 seconds\n",
      "[proc 1]sample: 66.840, forward: 150.198, backward: 51.866, update: 109.533\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.36332006369680164\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.3634053598716855\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.48117218354344365\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.48063534608185293\n",
      "[proc 1][Train](80000/100000) average loss: 0.42224612367004155\n",
      "[proc 1][Train](80000/100000) average regularization: 0.037113028050214054\n",
      "[proc 0][Train](80000/100000) average loss: 0.42202035288512707\n",
      "[proc 1][Train] 20000 steps take 396.011 seconds\n",
      "[proc 1]sample: 65.413, forward: 151.595, backward: 47.890, update: 86.306\n",
      "[proc 0][Train](80000/100000) average regularization: 0.03712603466641158\n",
      "[proc 0][Train] 20000 steps take 396.024 seconds\n",
      "[proc 0]sample: 67.945, forward: 151.903, backward: 50.977, update: 124.903\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.3579979382902384\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.35802228550165893\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.4763679586827755\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.4765416070103645\n",
      "[proc 0][Train](100000/100000) average loss: 0.41718294847905635\n",
      "[proc 1][Train](100000/100000) average loss: 0.4172819461330771\n",
      "[proc 0][Train](100000/100000) average regularization: 0.03759809280373156\n",
      "[proc 1][Train](100000/100000) average regularization: 0.03758740549460053\n",
      "[proc 0][Train] 20000 steps take 363.836 seconds\n",
      "[proc 0]sample: 67.995, forward: 155.563, backward: 50.900, update: 89.089\n",
      "[proc 1][Train] 20000 steps take 363.847 seconds\n",
      "[proc 1]sample: 65.019, forward: 153.305, backward: 47.609, update: 87.594\n",
      "[1]Valid average MRR: 0.42433107201796333\n",
      "[1]Valid average MR: 96.64075597349803\n",
      "[1]Valid average HITS@1: 0.3310737656359588\n",
      "[1]Valid average HITS@3: 0.46145910647773003\n",
      "[1]Valid average HITS@10: 0.6027462088970904\n",
      "[proc 1]validation take 378.672 seconds:\n",
      "[0]Valid average MRR: 0.42388170415212406\n",
      "[0]Valid average MR: 98.12352236204173\n",
      "[0]Valid average HITS@1: 0.33019079915018795\n",
      "[0]Valid average HITS@3: 0.46102644767663564\n",
      "[0]Valid average HITS@10: 0.6025153892248188\n",
      "[proc 0]validation take 400.280 seconds:\n",
      "proc 0 takes 2716.780 seconds\n",
      "proc 1 takes 2716.778 seconds\n",
      "training takes 2717.3132972717285 seconds\n",
      "Save model to ckpts/DistMult_DRKG_8\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.42360029906036184\n",
      "Test average MR : 97.24366049966974\n",
      "Test average HITS@1 : 0.32979360874864666\n",
      "Test average HITS@3 : 0.4610879971673124\n",
      "Test average HITS@10 : 0.60255214256045\n",
      "-----------------------------------------\n",
      "testing takes 385.248 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 200 \\\n",
    "--gamma 200.0 --lr 0.1 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: 200, **400**\n",
    "\n",
    "- gamma: **50**, 125, 200\n",
    "\n",
    "- lr: **0.01**, 0.05, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.431 seconds\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.4158593353509903\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.41543584139496087\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.5876080206394195\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.5878328572645783\n",
      "[proc 1][Train](20000/100000) average loss: 0.5017336778134107\n",
      "[proc 0][Train](20000/100000) average loss: 0.5016343495354056\n",
      "[proc 1][Train](20000/100000) average regularization: 0.020581230253275136\n",
      "[proc 0][Train](20000/100000) average regularization: 0.02059113002899685\n",
      "[proc 1][Train] 20000 steps take 624.818 seconds\n",
      "[proc 1]sample: 71.123, forward: 226.880, backward: 55.619, update: 269.496\n",
      "[proc 0][Train] 20000 steps take 624.853 seconds\n",
      "[proc 0]sample: 68.935, forward: 228.785, backward: 53.599, update: 204.204\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.43543543744683266\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.43535025097578767\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.5593202900663018\n",
      "[proc 0][Train](40000/100000) average loss: 0.4973352706596255\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.5590436056643724\n",
      "[proc 0][Train](40000/100000) average regularization: 0.021729698627255857\n",
      "[proc 0][Train] 20000 steps take 557.977 seconds\n",
      "[proc 0]sample: 68.400, forward: 239.547, backward: 51.398, update: 153.392\n",
      "[proc 1][Train](40000/100000) average loss: 0.4972395215958357\n",
      "[proc 1][Train](40000/100000) average regularization: 0.021728268125932662\n",
      "[proc 1][Train] 20000 steps take 557.989 seconds\n",
      "[proc 1]sample: 71.308, forward: 238.726, backward: 54.203, update: 192.539\n",
      "[1]Valid average MRR: 0.2648958103727455\n",
      "[1]Valid average MR: 130.5777593168865\n",
      "[1]Valid average HITS@1: 0.18648413082113893\n",
      "[1]Valid average HITS@3: 0.2834730383978973\n",
      "[1]Valid average HITS@10: 0.41359962412414797\n",
      "[proc 1]validation take 404.107 seconds:\n",
      "[0]Valid average MRR: 0.2645383775434489\n",
      "[0]Valid average MR: 131.90756591490984\n",
      "[0]Valid average HITS@1: 0.18580446151331917\n",
      "[0]Valid average HITS@3: 0.2838324617312197\n",
      "[0]Valid average HITS@10: 0.4136943672713406\n",
      "[proc 0]validation take 423.982 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.4434268231734633\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.4433534469142556\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.5484096902802587\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.5486586726352572\n",
      "[proc 1][Train](60000/100000) average loss: 0.4959182567164302\n",
      "[proc 0][Train](60000/100000) average loss: 0.4960060595929623\n",
      "[proc 1][Train](60000/100000) average regularization: 0.02186010717479512\n",
      "[proc 0][Train](60000/100000) average regularization: 0.021860900659952314\n",
      "[proc 1][Train] 20000 steps take 904.962 seconds\n",
      "[proc 1]sample: 69.475, forward: 238.003, backward: 52.967, update: 119.266\n",
      "[proc 0][Train] 20000 steps take 904.973 seconds\n",
      "[proc 0]sample: 66.794, forward: 243.321, backward: 50.459, update: 111.990\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.4480105463683605\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.4480908780694008\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.5455207510814071\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.5453384809166193\n",
      "[proc 0][Train](80000/100000) average loss: 0.49676564895808695\n",
      "[proc 1][Train](80000/100000) average loss: 0.4967146795988083\n",
      "[proc 0][Train](80000/100000) average regularization: 0.021601776901446282\n",
      "[proc 1][Train](80000/100000) average regularization: 0.021601613986585288\n",
      "[proc 0][Train] 20000 steps take 466.148 seconds\n",
      "[proc 0]sample: 65.741, forward: 239.199, backward: 50.295, update: 107.568\n",
      "[proc 1][Train] 20000 steps take 466.152 seconds\n",
      "[proc 1]sample: 68.455, forward: 231.727, backward: 52.744, update: 112.089\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.4497538121268153\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.44967482228428124\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.5441370820447803\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.5438976844206452\n",
      "[proc 0][Train](100000/100000) average loss: 0.49690595208108423\n",
      "[proc 1][Train](100000/100000) average loss: 0.49682574851512906\n",
      "[proc 0][Train](100000/100000) average regularization: 0.021697588736377657\n",
      "[proc 1][Train](100000/100000) average regularization: 0.021698938274383545\n",
      "[proc 0][Train] 20000 steps take 465.493 seconds\n",
      "[proc 0]sample: 65.401, forward: 239.008, backward: 50.094, update: 106.878\n",
      "[proc 1][Train] 20000 steps take 465.492 seconds\n",
      "[proc 1]sample: 68.393, forward: 231.644, backward: 52.613, update: 112.257\n",
      "[1]Valid average MRR: 0.28283639816706024\n",
      "[1]Valid average MR: 117.26207126660628\n",
      "[1]Valid average HITS@1: 0.1986422165780317\n",
      "[1]Valid average HITS@3: 0.30610389698822665\n",
      "[1]Valid average HITS@10: 0.44374799975486356\n",
      "[proc 1]validation take 395.859 seconds:\n",
      "[0]Valid average MRR: 0.2831915304432085\n",
      "[0]Valid average MR: 118.44005692651305\n",
      "[0]Valid average HITS@1: 0.19956624176063628\n",
      "[0]Valid average HITS@3: 0.3056837990957128\n",
      "[0]Valid average HITS@10: 0.44326414991556357\n",
      "[proc 0]validation take 417.266 seconds:\n",
      "proc 0 takes 3436.711 seconds\n",
      "proc 1 takes 3436.679 seconds\n",
      "training takes 3437.2352151870728 seconds\n",
      "Save model to ckpts/DistMult_DRKG_9\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.28164023773855135\n",
      "Test average MR : 118.9227513839994\n",
      "Test average HITS@1 : 0.19801405448837986\n",
      "Test average HITS@3 : 0.30411386586951933\n",
      "Test average HITS@10 : 0.441468571467482\n",
      "-----------------------------------------\n",
      "testing takes 445.912 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 400 \\\n",
    "--gamma 50.0 --lr 0.01 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: 200, **400**\n",
    "\n",
    "- gamma: **50**, 125, 200\n",
    "\n",
    "- lr: 0.01, **0.05**, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.483 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.383358545742929\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.383545296856761\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.5061397173285485\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.5061507267206907\n",
      "[proc 0][Train](20000/100000) average loss: 0.4447491315081716\n",
      "[proc 1][Train](20000/100000) average loss: 0.44484801180809735\n",
      "[proc 1][Train](20000/100000) average regularization: 0.02617844171752222\n",
      "[proc 0][Train](20000/100000) average regularization: 0.02617931953517691\n",
      "[proc 1][Train] 20000 steps take 485.211 seconds\n",
      "[proc 1]sample: 69.725, forward: 240.174, backward: 52.938, update: 120.894\n",
      "[proc 0][Train] 20000 steps take 485.238 seconds\n",
      "[proc 0]sample: 66.919, forward: 246.512, backward: 49.162, update: 114.716\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.3546465529948473\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.3541576307862997\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.47282961054742334\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.47285130750387905\n",
      "[proc 1][Train](40000/100000) average loss: 0.4137380817577243\n",
      "[proc 0][Train](40000/100000) average loss: 0.41350446922034023\n",
      "[proc 1][Train](40000/100000) average regularization: 0.032861308344639836\n",
      "[proc 0][Train](40000/100000) average regularization: 0.03286177157238126\n",
      "[proc 1][Train] 20000 steps take 488.856 seconds\n",
      "[proc 1]sample: 70.869, forward: 240.918, backward: 53.549, update: 122.342\n",
      "[proc 0][Train] 20000 steps take 488.857 seconds\n",
      "[proc 0]sample: 67.710, forward: 246.008, backward: 48.924, update: 111.349\n",
      "[1]Valid average MRR: 0.4473875745817948\n",
      "[1]Valid average MR: 70.42474652212697\n",
      "[1]Valid average HITS@1: 0.3513043300625779\n",
      "[1]Valid average HITS@3: 0.486449403160898\n",
      "[1]Valid average HITS@10: 0.6301504184342592\n",
      "[proc 1]validation take 399.899 seconds:\n",
      "[0]Valid average MRR: 0.4478824879689027\n",
      "[0]Valid average MR: 70.1048510105137\n",
      "[0]Valid average HITS@1: 0.35194680503350223\n",
      "[0]Valid average HITS@3: 0.4868476602930762\n",
      "[0]Valid average HITS@10: 0.6295112218772131\n",
      "[proc 0]validation take 421.590 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.3413294951662421\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.34089524551182987\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.46136594410389664\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.46115167751163244\n",
      "[proc 1][Train](60000/100000) average loss: 0.40134771965295074\n",
      "[proc 1][Train](60000/100000) average regularization: 0.03519177584592253\n",
      "[proc 0][Train](60000/100000) average loss: 0.4010234615266323\n",
      "[proc 1][Train] 20000 steps take 927.877 seconds\n",
      "[proc 1]sample: 70.651, forward: 231.294, backward: 51.982, update: 125.304\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03519560846053064\n",
      "[proc 0][Train] 20000 steps take 927.883 seconds\n",
      "[proc 0]sample: 68.801, forward: 237.982, backward: 51.876, update: 145.573\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.33330140779167416\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.33290679346024993\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.45466711743175986\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.4547537737339735\n",
      "[proc 1][Train](80000/100000) average loss: 0.39398426256775854\n",
      "[proc 0][Train](80000/100000) average loss: 0.3938302835658193\n",
      "[proc 1][Train](80000/100000) average regularization: 0.03650447749234736\n",
      "[proc 0][Train](80000/100000) average regularization: 0.03650642232652754\n",
      "[proc 1][Train] 20000 steps take 486.460 seconds\n",
      "[proc 1]sample: 70.783, forward: 239.096, backward: 52.226, update: 124.027\n",
      "[proc 0][Train] 20000 steps take 486.457 seconds\n",
      "[proc 0]sample: 67.362, forward: 241.000, backward: 48.848, update: 117.207\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.32753756399154665\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.3278620257139206\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.4503942622438073\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.45028020877689123\n",
      "[proc 0][Train](100000/100000) average loss: 0.3889659130498767\n",
      "[proc 1][Train](100000/100000) average loss: 0.3890711172521114\n",
      "[proc 0][Train](100000/100000) average regularization: 0.03741339877098799\n",
      "[proc 0][Train] 20000 steps take 505.489 seconds\n",
      "[proc 0]sample: 69.492, forward: 250.321, backward: 51.018, update: 128.325\n",
      "[proc 1][Train](100000/100000) average regularization: 0.03740990759637207\n",
      "[proc 1][Train] 20000 steps take 505.495 seconds\n",
      "[proc 1]sample: 69.825, forward: 248.422, backward: 50.653, update: 130.865\n",
      "[0]Valid average MRR: 0.46720313530023155\n",
      "[0]Valid average MR: 77.76997875469849\n",
      "[0]Valid average HITS@1: 0.37509533148117885\n",
      "[0]Valid average HITS@3: 0.5046201721414174\n",
      "[0]Valid average HITS@10: 0.6419690036498339\n",
      "[proc 0]validation take 389.102 seconds:\n",
      "[1]Valid average MRR: 0.46727195280358425\n",
      "[1]Valid average MR: 78.06577146475823\n",
      "[1]Valid average HITS@1: 0.3755558127974833\n",
      "[1]Valid average HITS@3: 0.5039698482196967\n",
      "[1]Valid average HITS@10: 0.6416990678006496\n",
      "[proc 1]validation take 395.989 seconds:\n",
      "proc 1 takes 3289.889 seconds\n",
      "proc 0 takes 3289.915 seconds\n",
      "training takes 3290.419363975525 seconds\n",
      "Save model to ckpts/DistMult_DRKG_10\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.4664626605382707\n",
      "Test average MR : 79.3194587251544\n",
      "Test average HITS@1 : 0.37422628815786785\n",
      "Test average HITS@3 : 0.5042830780963795\n",
      "Test average HITS@10 : 0.6413007211096509\n",
      "-----------------------------------------\n",
      "testing takes 400.930 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 400 \\\n",
    "--gamma 50.0 --lr 0.05 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 12\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: 200, **400**\n",
    "\n",
    "- gamma: **50**, 125, 200\n",
    "\n",
    "- lr: 0.01, 0.05, **0.1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.583 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.3405415379628539\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.340508358488977\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.48535354651361706\n",
      "[proc 0][Train](20000/100000) average loss: 0.4129475421488285\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.48885482498258354\n",
      "[proc 0][Train](20000/100000) average regularization: 0.03479059743471444\n",
      "[proc 0][Train] 20000 steps take 626.788 seconds\n",
      "[proc 0]sample: 71.626, forward: 244.257, backward: 55.004, update: 255.613\n",
      "[proc 1][Train](20000/100000) average loss: 0.41468159174621105\n",
      "[proc 1][Train](20000/100000) average regularization: 0.03474371984411846\n",
      "[proc 1][Train] 20000 steps take 626.746 seconds\n",
      "[proc 1]sample: 70.350, forward: 240.105, backward: 51.684, update: 128.247\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.31274803179949523\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.31283097455650566\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.44441634863317014\n",
      "[proc 0][Train](40000/100000) average loss: 0.37858219024837014\n",
      "[proc 0][Train](40000/100000) average regularization: 0.038652829253673554\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.4451763017177582\n",
      "[proc 0][Train] 20000 steps take 646.881 seconds\n",
      "[proc 0]sample: 70.933, forward: 232.379, backward: 53.927, update: 289.353\n",
      "[proc 1][Train](40000/100000) average loss: 0.37900363822579386\n",
      "[proc 1][Train](40000/100000) average regularization: 0.03863170288130641\n",
      "[proc 1][Train] 20000 steps take 646.886 seconds\n",
      "[proc 1]sample: 69.557, forward: 233.113, backward: 53.278, update: 121.420\n",
      "[0]Valid average MRR: 0.4797166513149649\n",
      "[0]Valid average MR: 95.39044369995098\n",
      "[0]Valid average HITS@1: 0.3929972217682628\n",
      "[0]Valid average HITS@3: 0.5137992319006374\n",
      "[0]Valid average HITS@10: 0.6458844037696791\n",
      "[proc 0]validation take 403.615 seconds:\n",
      "[1]Valid average MRR: 0.4796247245792885\n",
      "[1]Valid average MR: 94.66828615592038\n",
      "[1]Valid average HITS@1: 0.39311711392715365\n",
      "[1]Valid average HITS@3: 0.5133224837767352\n",
      "[1]Valid average HITS@10: 0.6451309777538694\n",
      "[proc 1]validation take 421.822 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.3072013983935118\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.3071739355549216\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.4395337889060378\n",
      "[proc 1][Train](60000/100000) average loss: 0.3733675936281681\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.43881483721882103\n",
      "[proc 1][Train](60000/100000) average regularization: 0.04014959220737219\n",
      "[proc 0][Train](60000/100000) average loss: 0.37299438651800154\n",
      "[proc 1][Train] 20000 steps take 1074.017 seconds\n",
      "[proc 1]sample: 69.252, forward: 230.608, backward: 52.573, update: 121.357\n",
      "[proc 0][Train](60000/100000) average regularization: 0.040159668450802564\n",
      "[proc 0][Train] 20000 steps take 1074.045 seconds\n",
      "[proc 0]sample: 70.622, forward: 224.521, backward: 54.518, update: 302.255\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.30420385540425776\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.3045908388152719\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.43615127432495354\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.43641636918634175\n",
      "[proc 0][Train](80000/100000) average loss: 0.3701775649115443\n",
      "[proc 1][Train](80000/100000) average loss: 0.37050360393077136\n",
      "[proc 1][Train](80000/100000) average regularization: 0.040998816468939187\n",
      "[proc 0][Train](80000/100000) average regularization: 0.04100369411334395\n",
      "[proc 1][Train] 20000 steps take 596.295 seconds\n",
      "[proc 1]sample: 70.207, forward: 233.452, backward: 50.860, update: 126.814\n",
      "[proc 0][Train] 20000 steps take 596.284 seconds\n",
      "[proc 0]sample: 71.548, forward: 235.915, backward: 54.766, update: 233.582\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.3023635051131248\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.3027963900193572\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.4343084849372506\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.4344882420510054\n",
      "[proc 1][Train](100000/100000) average loss: 0.3685524375140667\n",
      "[proc 0][Train](100000/100000) average loss: 0.3684258735612035\n",
      "[proc 1][Train](100000/100000) average regularization: 0.04155795702766627\n",
      "[proc 0][Train](100000/100000) average regularization: 0.04155815102253109\n",
      "[proc 1][Train] 20000 steps take 512.435 seconds\n",
      "[proc 1]sample: 70.822, forward: 243.573, backward: 51.638, update: 132.901\n",
      "[proc 0][Train] 20000 steps take 512.435 seconds\n",
      "[proc 0]sample: 71.614, forward: 255.933, backward: 53.662, update: 130.921\n",
      "[0]Valid average MRR: 0.48439492118073163\n",
      "[0]Valid average MR: 105.27600166149153\n",
      "[0]Valid average HITS@1: 0.40132170289262953\n",
      "[0]Valid average HITS@3: 0.5158250258756878\n",
      "[0]Valid average HITS@10: 0.6421630713079479\n",
      "[proc 0]validation take 383.534 seconds:\n",
      "[1]Valid average MRR: 0.48397887333135575\n",
      "[1]Valid average MR: 104.80246089733551\n",
      "[1]Valid average HITS@1: 0.40090019542820565\n",
      "[1]Valid average HITS@3: 0.5151031275322252\n",
      "[1]Valid average HITS@10: 0.642505975200365\n",
      "[proc 1]validation take 406.623 seconds:\n",
      "proc 1 takes 3863.003 seconds\n",
      "proc 0 takes 3863.056 seconds\n",
      "training takes 3863.5771803855896 seconds\n",
      "Save model to ckpts/DistMult_DRKG_11\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.4837632646888461\n",
      "Test average MR : 105.5472364272728\n",
      "Test average HITS@1 : 0.4006771893746978\n",
      "Test average HITS@3 : 0.51485118176185\n",
      "Test average HITS@10 : 0.6430064620685428\n",
      "-----------------------------------------\n",
      "testing takes 403.171 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 400 \\\n",
    "--gamma 50.0 --lr 0.1 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 13\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: 200, **400**\n",
    "\n",
    "- gamma: 50, **125**, 200\n",
    "\n",
    "- lr: **0.01**, 0.05, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.627 seconds\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.4095848518535495\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.40914382050037384\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.5988265433490276\n",
      "[proc 1][Train](20000/100000) average loss: 0.5042056974336505\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.5997970301076769\n",
      "[proc 1][Train](20000/100000) average regularization: 0.023655453068576752\n",
      "[proc 1][Train] 20000 steps take 606.669 seconds\n",
      "[proc 1]sample: 70.234, forward: 217.795, backward: 54.631, update: 263.666\n",
      "[proc 0][Train](20000/100000) average loss: 0.5044704253330827\n",
      "[proc 0][Train](20000/100000) average regularization: 0.023590294892608655\n",
      "[proc 0][Train] 20000 steps take 606.693 seconds\n",
      "[proc 0]sample: 68.124, forward: 223.994, backward: 51.625, update: 115.636\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.4060290642127395\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.4054800590351224\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.5545514211505651\n",
      "[proc 1][Train](40000/100000) average loss: 0.4802902425512671\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.5550267918497324\n",
      "[proc 1][Train](40000/100000) average regularization: 0.02838079589335248\n",
      "[proc 1][Train] 20000 steps take 630.944 seconds\n",
      "[proc 1]sample: 69.751, forward: 215.011, backward: 54.140, update: 291.697\n",
      "[proc 0][Train](40000/100000) average loss: 0.4802534254267812\n",
      "[proc 0][Train](40000/100000) average regularization: 0.028358798575215043\n",
      "[proc 0][Train] 20000 steps take 630.941 seconds\n",
      "[proc 0]sample: 68.498, forward: 216.214, backward: 51.627, update: 121.573\n",
      "[1]Valid average MRR: 0.2899624027062738\n",
      "[1]Valid average MR: 141.94776891806316\n",
      "[1]Valid average HITS@1: 0.20832851004718878\n",
      "[1]Valid average HITS@3: 0.3144998195523537\n",
      "[1]Valid average HITS@10: 0.4425155082835684\n",
      "[proc 1]validation take 365.867 seconds:\n",
      "[0]Valid average MRR: 0.28833572635679144\n",
      "[0]Valid average MR: 143.8260200468486\n",
      "[0]Valid average HITS@1: 0.2067195075448058\n",
      "[0]Valid average HITS@3: 0.3123706215612573\n",
      "[0]Valid average HITS@10: 0.44097959906302775\n",
      "[proc 0]validation take 368.792 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.40535909152776\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.4050188923791051\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.5314298865839839\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.5315793607354165\n",
      "[proc 1][Train](60000/100000) average loss: 0.46839448913931847\n",
      "[proc 1][Train](60000/100000) average regularization: 0.030270206131599843\n",
      "[proc 0][Train](60000/100000) average loss: 0.4682991266235709\n",
      "[proc 1][Train] 20000 steps take 929.890 seconds\n",
      "[proc 1]sample: 69.873, forward: 222.668, backward: 53.551, update: 214.470\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03026199292074889\n",
      "[proc 0][Train] 20000 steps take 929.881 seconds\n",
      "[proc 0]sample: 67.436, forward: 227.633, backward: 50.531, update: 130.379\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.40089059502929447\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.5179521310299635\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.4005870055735111\n",
      "[proc 1][Train](80000/100000) average loss: 0.4594213629752398\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.5180927079841494\n",
      "[proc 1][Train](80000/100000) average regularization: 0.03158574140165001\n",
      "[proc 1][Train] 20000 steps take 452.003 seconds\n",
      "[proc 1]sample: 67.337, forward: 223.173, backward: 51.567, update: 109.502\n",
      "[proc 0][Train](80000/100000) average loss: 0.45933985685259104\n",
      "[proc 0][Train](80000/100000) average regularization: 0.031583548494800925\n",
      "[proc 0][Train] 20000 steps take 452.005 seconds\n",
      "[proc 0]sample: 64.364, forward: 228.884, backward: 49.090, update: 104.690\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.39641297114044427\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.39614887129217385\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.5089628316551447\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.5089733736693859\n",
      "[proc 1][Train](100000/100000) average loss: 0.4526879014387727\n",
      "[proc 0][Train](100000/100000) average loss: 0.45256112240105867\n",
      "[proc 1][Train](100000/100000) average regularization: 0.03247953853476793\n",
      "[proc 0][Train](100000/100000) average regularization: 0.03247632688581944\n",
      "[proc 1][Train] 20000 steps take 454.813 seconds\n",
      "[proc 1]sample: 67.706, forward: 225.234, backward: 51.511, update: 110.029\n",
      "[proc 0][Train] 20000 steps take 454.808 seconds\n",
      "[proc 0]sample: 64.203, forward: 229.555, backward: 48.553, update: 103.985\n",
      "[0]Valid average MRR: 0.31773067910122527\n",
      "[0]Valid average MR: 117.0837418968241\n",
      "[0]Valid average HITS@1: 0.22975567903252164\n",
      "[0]Valid average HITS@3: 0.3454268126600207\n",
      "[0]Valid average HITS@10: 0.4856526120825843\n",
      "[proc 0]validation take 354.605 seconds:\n",
      "[1]Valid average MRR: 0.3190512146807909\n",
      "[1]Valid average MR: 115.66930755769218\n",
      "[1]Valid average HITS@1: 0.23084360977004842\n",
      "[1]Valid average HITS@3: 0.34666376134607135\n",
      "[1]Valid average HITS@10: 0.4870111741353834\n",
      "[proc 1]validation take 357.194 seconds:\n",
      "proc 1 takes 3431.513 seconds\n",
      "proc 0 takes 3431.520 seconds\n",
      "training takes 3432.0284922122955 seconds\n",
      "Save model to ckpts/DistMult_DRKG_12\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.3181590512743888\n",
      "Test average MR : 117.36893883165256\n",
      "Test average HITS@1 : 0.23009117713149527\n",
      "Test average HITS@3 : 0.3460458132741374\n",
      "Test average HITS@10 : 0.4859999863813097\n",
      "-----------------------------------------\n",
      "testing takes 383.510 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 400 \\\n",
    "--gamma 125.0 --lr 0.01 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: 200, **400**\n",
    "\n",
    "- gamma: 50, **125**, 200\n",
    "\n",
    "- lr: 0.01, **0.05**, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.706 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.37238707480877636\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.3717193614915013\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.4951714565634727\n",
      "[proc 0][Train](20000/100000) average loss: 0.43377926576137543\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.49749376702308656\n",
      "[proc 0][Train](20000/100000) average regularization: 0.029983778939163312\n",
      "[proc 1][Train](20000/100000) average loss: 0.43460656431913375\n",
      "[proc 0][Train] 20000 steps take 632.778 seconds\n",
      "[proc 0]sample: 70.859, forward: 231.346, backward: 52.323, update: 277.957\n",
      "[proc 1][Train](20000/100000) average regularization: 0.0298504261797294\n",
      "[proc 1][Train] 20000 steps take 632.769 seconds\n",
      "[proc 1]sample: 71.613, forward: 234.450, backward: 51.938, update: 121.994\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.34282766007184984\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.46182859376221896\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.34266397794038056\n",
      "[proc 0][Train](40000/100000) average loss: 0.4023281269103289\n",
      "[proc 0][Train](40000/100000) average regularization: 0.0360060906317085\n",
      "[proc 0][Train] 20000 steps take 655.829 seconds\n",
      "[proc 0]sample: 70.095, forward: 226.498, backward: 52.723, update: 306.217\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.4623400511994958\n",
      "[proc 1][Train](40000/100000) average loss: 0.40250201464146373\n",
      "[proc 1][Train](40000/100000) average regularization: 0.03598570260647684\n",
      "[proc 1][Train] 20000 steps take 655.837 seconds\n",
      "[proc 1]sample: 70.631, forward: 231.563, backward: 51.020, update: 121.063\n",
      "[0]Valid average MRR: 0.4501687721888757\n",
      "[0]Valid average MR: 80.04412145230702\n",
      "[0]Valid average HITS@1: 0.35852467723484227\n",
      "[0]Valid average HITS@3: 0.48652080950046306\n",
      "[0]Valid average HITS@10: 0.6242475622378384\n",
      "[proc 0]validation take 414.957 seconds:\n",
      "[1]Valid average MRR: 0.4498231958663024\n",
      "[1]Valid average MR: 77.88163655801222\n",
      "[1]Valid average HITS@1: 0.35819879202217125\n",
      "[1]Valid average HITS@3: 0.48594891629272013\n",
      "[1]Valid average HITS@10: 0.6243454516979102\n",
      "[proc 1]validation take 418.844 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.3322633984327316\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.33218044827878473\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.45295916556715965\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.4534470525562763\n",
      "[proc 0][Train](60000/100000) average loss: 0.3925698068663478\n",
      "[proc 1][Train](60000/100000) average loss: 0.39285522551089525\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03768210592661053\n",
      "[proc 1][Train](60000/100000) average regularization: 0.03767226090207696\n",
      "[proc 0][Train] 20000 steps take 1030.507 seconds\n",
      "[proc 0]sample: 68.566, forward: 227.814, backward: 51.333, update: 262.105\n",
      "[proc 1][Train] 20000 steps take 1030.489 seconds\n",
      "[proc 1]sample: 69.344, forward: 229.263, backward: 51.553, update: 118.528\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.32633329341858625\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.32605119233578445\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.4484642797097564\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.4484077033877373\n",
      "[proc 1][Train](80000/100000) average loss: 0.3873987865269184\n",
      "[proc 1][Train](80000/100000) average regularization: 0.0386363993993029\n",
      "[proc 0][Train](80000/100000) average loss: 0.3872294478997588\n",
      "[proc 1][Train] 20000 steps take 468.936 seconds\n",
      "[proc 1]sample: 69.179, forward: 231.953, backward: 52.740, update: 114.782\n",
      "[proc 0][Train](80000/100000) average regularization: 0.03863692744653672\n",
      "[proc 0][Train] 20000 steps take 468.941 seconds\n",
      "[proc 0]sample: 65.725, forward: 237.574, backward: 48.744, update: 105.412\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.3221035138219595\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.32225737878233196\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.4452217407315969\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.4453364287734032\n",
      "[proc 0][Train](100000/100000) average loss: 0.3836626272588968\n",
      "[proc 1][Train](100000/100000) average loss: 0.3837969037145376\n",
      "[proc 0][Train](100000/100000) average regularization: 0.03931381965503097\n",
      "[proc 1][Train](100000/100000) average regularization: 0.039315260035544634\n",
      "[proc 0][Train] 20000 steps take 472.082 seconds\n",
      "[proc 0]sample: 65.330, forward: 237.237, backward: 48.843, update: 105.143\n",
      "[proc 1][Train] 20000 steps take 472.089 seconds\n",
      "[proc 1]sample: 69.806, forward: 233.339, backward: 52.627, update: 116.021\n",
      "[1]Valid average MRR: 0.4672355973698691\n",
      "[1]Valid average MR: 85.05888721681636\n",
      "[1]Valid average HITS@1: 0.37867449287402033\n",
      "[1]Valid average HITS@3: 0.5021279203578992\n",
      "[1]Valid average HITS@10: 0.6359519804980355\n",
      "[proc 1]validation take 405.163 seconds:\n",
      "[0]Valid average MRR: 0.46757057693778614\n",
      "[0]Valid average MR: 87.67238315084164\n",
      "[0]Valid average HITS@1: 0.3790822302119083\n",
      "[0]Valid average HITS@3: 0.5022641226779975\n",
      "[0]Valid average HITS@10: 0.6359154545949774\n",
      "[proc 0]validation take 411.251 seconds:\n",
      "proc 0 takes 3671.389 seconds\n",
      "proc 1 takes 3671.369 seconds\n",
      "training takes 3671.904219150543 seconds\n",
      "Save model to ckpts/DistMult_DRKG_13\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.46677116851926154\n",
      "Test average MR : 87.42674166025454\n",
      "Test average HITS@1 : 0.37791184621774926\n",
      "Test average HITS@3 : 0.5020428035435832\n",
      "Test average HITS@10 : 0.6359145290997364\n",
      "-----------------------------------------\n",
      "testing takes 408.111 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 400 \\\n",
    "--gamma 125.0 --lr 0.05 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 15\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: 200, **400**\n",
    "\n",
    "- gamma: 50, **125**, 200\n",
    "\n",
    "- lr: 0.01, 0.05, **0.1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.310 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.3483994608566165\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.34877421112954615\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.493459678901732\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.49351727146953345\n",
      "[proc 0][Train](20000/100000) average loss: 0.4209295699775219\n",
      "[proc 1][Train](20000/100000) average loss: 0.421145741315186\n",
      "[proc 0][Train](20000/100000) average regularization: 0.03380008188744541\n",
      "[proc 1][Train](20000/100000) average regularization: 0.03388066169428639\n",
      "[proc 0][Train] 20000 steps take 492.715 seconds\n",
      "[proc 0]sample: 69.655, forward: 242.307, backward: 52.241, update: 127.996\n",
      "[proc 1][Train] 20000 steps take 492.711 seconds\n",
      "[proc 1]sample: 67.637, forward: 237.796, backward: 50.172, update: 129.574\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.31663466578722\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.31631429158896207\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.4486838324397802\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.4485105546951294\n",
      "[proc 1][Train](40000/100000) average loss: 0.38257261028438805\n",
      "[proc 0][Train](40000/100000) average loss: 0.382499061948061\n",
      "[proc 1][Train](40000/100000) average regularization: 0.03870570112075657\n",
      "[proc 0][Train](40000/100000) average regularization: 0.038719748068787155\n",
      "[proc 1][Train] 20000 steps take 490.529 seconds\n",
      "[proc 1]sample: 67.550, forward: 235.653, backward: 49.004, update: 128.176\n",
      "[proc 0][Train] 20000 steps take 490.533 seconds\n",
      "[proc 0]sample: 70.611, forward: 242.342, backward: 52.535, update: 124.504\n",
      "[0]Valid average MRR: 0.4723437772271343\n",
      "[0]Valid average MR: 98.16282957454922\n",
      "[0]Valid average HITS@1: 0.385806504330773\n",
      "[0]Valid average HITS@3: 0.5054577272974887\n",
      "[0]Valid average HITS@10: 0.6381931960560004\n",
      "[proc 0]validation take 389.078 seconds:\n",
      "[1]Valid average MRR: 0.4730267283037608\n",
      "[1]Valid average MR: 96.76895891922074\n",
      "[1]Valid average HITS@1: 0.3869342285352418\n",
      "[1]Valid average HITS@3: 0.5050355107349326\n",
      "[1]Valid average HITS@10: 0.6392034428049055\n",
      "[proc 1]validation take 398.907 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.31004172273129227\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.3097575331225991\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.4418864328458905\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.44169363436102865\n",
      "[proc 1][Train](60000/100000) average loss: 0.37596407768428325\n",
      "[proc 0][Train](60000/100000) average loss: 0.3757255838111043\n",
      "[proc 1][Train](60000/100000) average regularization: 0.04020347325839102\n",
      "[proc 0][Train](60000/100000) average regularization: 0.040215861464105544\n",
      "[proc 0][Train] 20000 steps take 891.018 seconds\n",
      "[proc 0]sample: 69.112, forward: 243.895, backward: 52.884, update: 120.423\n",
      "[proc 1][Train] 20000 steps take 891.022 seconds\n",
      "[proc 1]sample: 69.532, forward: 240.119, backward: 51.287, update: 127.027\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.3064714848160744\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.4385167731747031\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.3067421342983842\n",
      "[proc 0][Train](80000/100000) average loss: 0.3724941290527582\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.4382648483484983\n",
      "[proc 0][Train](80000/100000) average regularization: 0.041052515325695274\n",
      "[proc 0][Train] 20000 steps take 500.370 seconds\n",
      "[proc 0]sample: 70.982, forward: 247.816, backward: 50.776, update: 121.958\n",
      "[proc 1][Train](80000/100000) average loss: 0.372503491294384\n",
      "[proc 1][Train](80000/100000) average regularization: 0.0410396696863696\n",
      "[proc 1][Train] 20000 steps take 500.380 seconds\n",
      "[proc 1]sample: 70.956, forward: 245.655, backward: 53.903, update: 129.074\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.3044052132681012\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.30454876629859207\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.43616709588319064\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.4360867990806699\n",
      "[proc 0][Train](100000/100000) average loss: 0.37028615456819536\n",
      "[proc 0][Train](100000/100000) average regularization: 0.041611099437065424\n",
      "[proc 1][Train](100000/100000) average loss: 0.3703177826672792\n",
      "[proc 0][Train] 20000 steps take 495.961 seconds\n",
      "[proc 0]sample: 69.258, forward: 249.152, backward: 50.374, update: 124.637\n",
      "[proc 1][Train](100000/100000) average regularization: 0.041597075597010556\n",
      "[proc 1][Train] 20000 steps take 495.952 seconds\n",
      "[proc 1]sample: 70.731, forward: 242.176, backward: 54.016, update: 128.002\n",
      "[1]Valid average MRR: 0.4799116844790812\n",
      "[1]Valid average MR: 104.82799594163029\n",
      "[1]Valid average HITS@1: 0.3963958136146047\n",
      "[1]Valid average HITS@3: 0.510666839170077\n",
      "[1]Valid average HITS@10: 0.6406572379934222\n",
      "[proc 1]validation take 374.352 seconds:\n",
      "[0]Valid average MRR: 0.4796667773895746\n",
      "[0]Valid average MR: 105.90874734433731\n",
      "[0]Valid average HITS@1: 0.3962214686495615\n",
      "[0]Valid average HITS@3: 0.5108984311161955\n",
      "[0]Valid average HITS@10: 0.6391703437380836\n",
      "[proc 0]validation take 380.768 seconds:\n",
      "proc 0 takes 3251.366 seconds\n",
      "proc 1 takes 3251.359 seconds\n",
      "training takes 3251.9182755947113 seconds\n",
      "Save model to ckpts/DistMult_DRKG_14\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.47940834784000336\n",
      "Test average MR : 107.53529624056054\n",
      "Test average HITS@1 : 0.39559912023260724\n",
      "Test average HITS@3 : 0.5107604676658246\n",
      "Test average HITS@10 : 0.6406691543474264\n",
      "-----------------------------------------\n",
      "testing takes 383.557 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 400 \\\n",
    "--gamma 125.0 --lr 0.1 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 16\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: 200, **400**\n",
    "\n",
    "- gamma: 50, 125, **200**\n",
    "\n",
    "- lr: **0.01**, 0.05, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.444 seconds\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.41796293351352215\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.417397934910655\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.6264719388902187\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.6264910311222076\n",
      "[proc 1][Train](20000/100000) average loss: 0.522217436374724\n",
      "[proc 1][Train](20000/100000) average regularization: 0.0276744533224497\n",
      "[proc 0][Train](20000/100000) average loss: 0.5219444829180837\n",
      "[proc 1][Train] 20000 steps take 519.393 seconds\n",
      "[proc 1]sample: 72.909, forward: 259.388, backward: 54.352, update: 130.293\n",
      "[proc 0][Train](20000/100000) average regularization: 0.027677741696313023\n",
      "[proc 0][Train] 20000 steps take 519.417 seconds\n",
      "[proc 0]sample: 69.672, forward: 258.241, backward: 53.102, update: 135.741\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.417327788297832\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.4179550715625286\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.5655258344218135\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.5655188549339771\n",
      "[proc 0][Train](40000/100000) average loss: 0.49142681132405996\n",
      "[proc 0][Train](40000/100000) average regularization: 0.03249429781548679\n",
      "[proc 1][Train](40000/100000) average loss: 0.49173696317672727\n",
      "[proc 0][Train] 20000 steps take 518.172 seconds\n",
      "[proc 0]sample: 70.873, forward: 240.022, backward: 53.035, update: 142.413\n",
      "[proc 1][Train](40000/100000) average regularization: 0.032490019525401294\n",
      "[proc 1][Train] 20000 steps take 518.190 seconds\n",
      "[proc 1]sample: 70.014, forward: 222.888, backward: 50.683, update: 155.871\n",
      "[1]Valid average MRR: 0.2703521212253118\n",
      "[1]Valid average MR: 144.75398857391883\n",
      "[1]Valid average HITS@1: 0.19247635454898301\n",
      "[1]Valid average HITS@3: 0.2912356918635135\n",
      "[1]Valid average HITS@10: 0.4169464172630518\n",
      "[proc 1]validation take 359.343 seconds:\n",
      "[0]Valid average MRR: 0.2696217839687559\n",
      "[0]Valid average MR: 145.27232118537887\n",
      "[0]Valid average HITS@1: 0.19185460587241923\n",
      "[0]Valid average HITS@3: 0.29033883532167565\n",
      "[0]Valid average HITS@10: 0.41570654246336547\n",
      "[proc 0]validation take 384.873 seconds:\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.41998906807005404\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.4204980254724622\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.5342870845675468\n",
      "[proc 0][Train](60000/100000) average loss: 0.47713807634711264\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.5344565542846919\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03292998957727104\n",
      "[proc 1][Train](60000/100000) average loss: 0.4774772899463773\n",
      "[proc 0][Train] 20000 steps take 1024.848 seconds\n",
      "[proc 0]sample: 70.186, forward: 225.817, backward: 55.196, update: 288.477\n",
      "[proc 1][Train](60000/100000) average regularization: 0.032927879514917734\n",
      "[proc 1][Train] 20000 steps take 1024.846 seconds\n",
      "[proc 1]sample: 70.751, forward: 217.766, backward: 52.914, update: 210.912\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.4205186055928469\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.5236160859778524\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.42111985391378404\n",
      "[proc 0][Train](80000/100000) average loss: 0.4720673458904028\n",
      "[proc 0][Train](80000/100000) average regularization: 0.0325407257810235\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.5235937897235156\n",
      "[proc 0][Train] 20000 steps take 654.006 seconds\n",
      "[proc 0]sample: 70.531, forward: 228.588, backward: 54.797, update: 299.791\n",
      "[proc 1][Train](80000/100000) average loss: 0.4723568217560649\n",
      "[proc 1][Train](80000/100000) average regularization: 0.03253676782753318\n",
      "[proc 1][Train] 20000 steps take 654.018 seconds\n",
      "[proc 1]sample: 71.573, forward: 224.923, backward: 53.768, update: 282.266\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.41805857417434455\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.4185501248180866\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.5188197800070047\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.5187990230724215\n",
      "[proc 0][Train](100000/100000) average loss: 0.4684391771301627\n",
      "[proc 0][Train](100000/100000) average regularization: 0.03265907969865948\n",
      "[proc 1][Train](100000/100000) average loss: 0.46867457390725614\n",
      "[proc 0][Train] 20000 steps take 633.862 seconds\n",
      "[proc 0]sample: 70.822, forward: 229.364, backward: 54.451, update: 278.922\n",
      "[proc 1][Train](100000/100000) average regularization: 0.0326532020509243\n",
      "[proc 1][Train] 20000 steps take 633.846 seconds\n",
      "[proc 1]sample: 67.974, forward: 236.243, backward: 53.026, update: 168.133\n",
      "[1]Valid average MRR: 0.2960559938766072\n",
      "[1]Valid average MR: 120.87413606433469\n",
      "[1]Valid average HITS@1: 0.2134014721804204\n",
      "[1]Valid average HITS@3: 0.32025712087268565\n",
      "[1]Valid average HITS@10: 0.4523822493990753\n",
      "[proc 1]validation take 346.219 seconds:\n",
      "[0]Valid average MRR: 0.29567337374799535\n",
      "[0]Valid average MR: 121.14874434820504\n",
      "[0]Valid average HITS@1: 0.2135595413193877\n",
      "[0]Valid average HITS@3: 0.3192072506400828\n",
      "[0]Valid average HITS@10: 0.4514183962521109\n",
      "[proc 0]validation take 360.924 seconds:\n",
      "proc 0 takes 3711.230 seconds\n",
      "proc 1 takes 3711.215 seconds\n",
      "training takes 3711.738737821579 seconds\n",
      "Save model to ckpts/DistMult_DRKG_15\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.29492594865669725\n",
      "Test average MR : 122.09175252116005\n",
      "Test average HITS@1 : 0.2122217531340011\n",
      "Test average HITS@3 : 0.3191557092954371\n",
      "Test average HITS@10 : 0.4508961098211185\n",
      "-----------------------------------------\n",
      "testing takes 390.880 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 400 \\\n",
    "--gamma 200.0 --lr 0.01 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 17\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: 200, **400**\n",
    "\n",
    "- gamma: 50, 125, **200**\n",
    "\n",
    "- lr: 0.01, **0.05**, 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.350 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.39820926177799704\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.3997192317456007\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.5146068893954158\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.514714073894918\n",
      "[proc 0][Train](20000/100000) average loss: 0.45640807555168866\n",
      "[proc 0][Train](20000/100000) average regularization: 0.031306551433261484\n",
      "[proc 1][Train](20000/100000) average loss: 0.45721665294617414\n",
      "[proc 0][Train] 20000 steps take 495.305 seconds\n",
      "[proc 0]sample: 69.624, forward: 238.967, backward: 50.915, update: 134.523\n",
      "[proc 1][Train](20000/100000) average regularization: 0.031314672961318867\n",
      "[proc 1][Train] 20000 steps take 495.311 seconds\n",
      "[proc 1]sample: 70.057, forward: 239.329, backward: 51.096, update: 125.280\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.3604959198907018\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.3591701184347272\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.4751899610161781\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.4752137802064419\n",
      "[proc 1][Train](40000/100000) average loss: 0.4178429405793548\n",
      "[proc 0][Train](40000/100000) average loss: 0.4171919493019581\n",
      "[proc 1][Train](40000/100000) average regularization: 0.0375112090786919\n",
      "[proc 0][Train](40000/100000) average regularization: 0.03750605020392686\n",
      "[proc 0][Train] 20000 steps take 497.052 seconds\n",
      "[proc 0]sample: 70.543, forward: 236.739, backward: 51.304, update: 131.946\n",
      "[proc 1][Train] 20000 steps take 497.044 seconds\n",
      "[proc 1]sample: 70.017, forward: 224.953, backward: 49.756, update: 134.941\n",
      "[1]Valid average MRR: 0.4254970176628301\n",
      "[1]Valid average MR: 91.2677945212009\n",
      "[1]Valid average HITS@1: 0.3352989642986034\n",
      "[1]Valid average HITS@3: 0.4587047263664653\n",
      "[1]Valid average HITS@10: 0.5988614774917096\n",
      "[proc 1]validation take 405.493 seconds:\n",
      "[0]Valid average MRR: 0.4263312140332643\n",
      "[0]Valid average MR: 90.58504589529879\n",
      "[0]Valid average HITS@1: 0.33623072942201887\n",
      "[0]Valid average HITS@3: 0.4596986163316446\n",
      "[0]Valid average HITS@10: 0.6000878411505148\n",
      "[proc 0]validation take 412.853 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.34611501959860325\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.34506890789717437\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.4627803060427308\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.46313233982026575\n",
      "[proc 1][Train](60000/100000) average loss: 0.40444766269922255\n",
      "[proc 0][Train](60000/100000) average loss: 0.404100623857975\n",
      "[proc 1][Train](60000/100000) average regularization: 0.03923567107785493\n",
      "[proc 0][Train](60000/100000) average regularization: 0.03922467266395688\n",
      "[proc 1][Train] 20000 steps take 908.414 seconds\n",
      "[proc 1]sample: 70.757, forward: 232.546, backward: 50.553, update: 141.173\n",
      "[proc 0][Train] 20000 steps take 908.418 seconds\n",
      "[proc 0]sample: 68.031, forward: 236.222, backward: 50.804, update: 131.902\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.33750500885695217\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.338322888456285\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.45684453095942734\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.4565377616718411\n",
      "[proc 0][Train](80000/100000) average loss: 0.39717476979643107\n",
      "[proc 0][Train](80000/100000) average regularization: 0.04003875047173351\n",
      "[proc 1][Train](80000/100000) average loss: 0.3974303251594305\n",
      "[proc 0][Train] 20000 steps take 482.385 seconds\n",
      "[proc 0]sample: 67.252, forward: 233.009, backward: 50.875, update: 127.818\n",
      "[proc 1][Train](80000/100000) average regularization: 0.04004848876800388\n",
      "[proc 1][Train] 20000 steps take 482.398 seconds\n",
      "[proc 1]sample: 68.319, forward: 222.534, backward: 49.017, update: 141.799\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.3325191074773669\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.45279109587818384\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.3332042684286833\n",
      "[proc 0][Train](100000/100000) average loss: 0.3926551017135382\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.45251754746586087\n",
      "[proc 0][Train](100000/100000) average regularization: 0.040554650882445274\n",
      "[proc 1][Train](100000/100000) average loss: 0.3928609078839421\n",
      "[proc 0][Train] 20000 steps take 478.837 seconds\n",
      "[proc 0]sample: 66.629, forward: 230.073, backward: 51.060, update: 128.423\n",
      "[proc 1][Train](100000/100000) average regularization: 0.04056369966603816\n",
      "[proc 1][Train] 20000 steps take 478.831 seconds\n",
      "[proc 1]sample: 67.646, forward: 222.101, backward: 49.081, update: 139.288\n",
      "[1]Valid average MRR: 0.4520698517084259\n",
      "[1]Valid average MR: 96.14584595899413\n",
      "[1]Valid average HITS@1: 0.3637824550412987\n",
      "[1]Valid average HITS@3: 0.4851351995478595\n",
      "[1]Valid average HITS@10: 0.6227112088630437\n",
      "[proc 1]validation take 387.497 seconds:\n",
      "[0]Valid average MRR: 0.4533569558370097\n",
      "[0]Valid average MR: 95.54875864792722\n",
      "[0]Valid average HITS@1: 0.36535449692215505\n",
      "[0]Valid average HITS@3: 0.4866331644604238\n",
      "[0]Valid average HITS@10: 0.621860870512611\n",
      "[proc 0]validation take 415.723 seconds:\n",
      "proc 0 takes 3277.719 seconds\n",
      "proc 1 takes 3277.718 seconds\n",
      "training takes 3278.265002012253 seconds\n",
      "Save model to ckpts/DistMult_DRKG_16\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.4522995042699474\n",
      "Test average MR : 98.98765976426047\n",
      "Test average HITS@1 : 0.36402929380281496\n",
      "Test average HITS@3 : 0.48605105646989927\n",
      "Test average HITS@10 : 0.6215178711263338\n",
      "-----------------------------------------\n",
      "testing takes 442.001 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 400 \\\n",
    "--gamma 200.0 --lr 0.05 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 18\n",
    "\n",
    "- batch_size: **4096**\n",
    "\n",
    "- neg_sample_size: **256**\n",
    "\n",
    "- hidden_dim: 200, **400**\n",
    "\n",
    "- gamma: 50, 125, **200**\n",
    "\n",
    "- lr: 0.01, 0.05, **0.1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading train triples....\n",
      "Finished. Read 5286834 train triples.\n",
      "Reading valid triples....\n",
      "Finished. Read 293713 valid triples.\n",
      "Reading test triples....\n",
      "Finished. Read 293714 test triples.\n",
      "|Train|: 5286834\n",
      "random partition 5286834 edges into 2 parts\n",
      "part 0 has 2643417 edges\n",
      "part 1 has 2643417 edges\n",
      "/home/luyanfeng/miniconda3/envs/drkg/lib/python3.8/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.\n",
      "  warnings.warn(msg, warn_type)\n",
      "|valid|: 293713\n",
      "|test|: 293714\n",
      "Total initialize time 16.318 seconds\n",
      "[proc 0][Train](20000/100000) average pos_loss: 0.3673332645520568\n",
      "[proc 1][Train](20000/100000) average pos_loss: 0.3676285136580467\n",
      "[proc 0][Train](20000/100000) average neg_loss: 0.5051938116028905\n",
      "[proc 1][Train](20000/100000) average neg_loss: 0.5057804387018084\n",
      "[proc 0][Train](20000/100000) average loss: 0.43626353810876606\n",
      "[proc 1][Train](20000/100000) average loss: 0.43670447624772785\n",
      "[proc 0][Train](20000/100000) average regularization: 0.034539960472658275\n",
      "[proc 1][Train](20000/100000) average regularization: 0.034630826802179215\n",
      "[proc 0][Train] 20000 steps take 491.451 seconds\n",
      "[proc 0]sample: 70.626, forward: 236.727, backward: 51.326, update: 132.342\n",
      "[proc 1][Train] 20000 steps take 491.441 seconds\n",
      "[proc 1]sample: 68.827, forward: 233.122, backward: 51.171, update: 120.107\n",
      "[proc 1][Train](40000/100000) average pos_loss: 0.3261994378611445\n",
      "[proc 0][Train](40000/100000) average pos_loss: 0.3260754798680544\n",
      "[proc 1][Train](40000/100000) average neg_loss: 0.45539849907010793\n",
      "[proc 1][Train](40000/100000) average loss: 0.39079896841347217\n",
      "[proc 0][Train](40000/100000) average neg_loss: 0.455552515116334\n",
      "[proc 1][Train](40000/100000) average regularization: 0.03949530223850161\n",
      "[proc 0][Train](40000/100000) average loss: 0.390813997516036\n",
      "[proc 1][Train] 20000 steps take 480.745 seconds\n",
      "[proc 1]sample: 68.494, forward: 229.555, backward: 51.552, update: 117.497\n",
      "[proc 0][Train](40000/100000) average regularization: 0.0394829955983907\n",
      "[proc 0][Train] 20000 steps take 480.755 seconds\n",
      "[proc 0]sample: 70.842, forward: 228.198, backward: 50.862, update: 127.357\n",
      "[0]Valid average MRR: 0.4634334052050157\n",
      "[0]Valid average MR: 103.29771340633\n",
      "[0]Valid average HITS@1: 0.3752383287029471\n",
      "[0]Valid average HITS@3: 0.4981103938552051\n",
      "[0]Valid average HITS@10: 0.6318876995151713\n",
      "[proc 0]validation take 402.865 seconds:\n",
      "[1]Valid average MRR: 0.46325423165978885\n",
      "[1]Valid average MR: 102.34909469756293\n",
      "[1]Valid average HITS@1: 0.3756273109215087\n",
      "[1]Valid average HITS@3: 0.49739202080935874\n",
      "[1]Valid average HITS@10: 0.6310220146128547\n",
      "[proc 1]validation take 411.609 seconds:\n",
      "[proc 1][Train](60000/100000) average pos_loss: 0.3162036632731557\n",
      "[proc 0][Train](60000/100000) average pos_loss: 0.3161381689295173\n",
      "[proc 1][Train](60000/100000) average neg_loss: 0.4463601421251893\n",
      "[proc 0][Train](60000/100000) average neg_loss: 0.44651478747725487\n",
      "[proc 1][Train](60000/100000) average loss: 0.3812819026961923\n",
      "[proc 1][Train](60000/100000) average regularization: 0.04080369385667145\n",
      "[proc 0][Train](60000/100000) average loss: 0.3813264782637358\n",
      "[proc 1][Train] 20000 steps take 878.793 seconds\n",
      "[proc 1]sample: 67.313, forward: 227.013, backward: 51.221, update: 111.828\n",
      "[proc 0][Train](60000/100000) average regularization: 0.04078823979943991\n",
      "[proc 0][Train] 20000 steps take 878.791 seconds\n",
      "[proc 0]sample: 69.265, forward: 221.912, backward: 50.718, update: 120.305\n",
      "[proc 1][Train](80000/100000) average pos_loss: 0.311681779576838\n",
      "[proc 0][Train](80000/100000) average pos_loss: 0.3116729593411088\n",
      "[proc 1][Train](80000/100000) average neg_loss: 0.442030743290484\n",
      "[proc 0][Train](80000/100000) average neg_loss: 0.4420432328015566\n",
      "[proc 1][Train](80000/100000) average loss: 0.37685626143813133\n",
      "[proc 0][Train](80000/100000) average loss: 0.37685809619277716\n",
      "[proc 1][Train](80000/100000) average regularization: 0.041504114363528785\n",
      "[proc 0][Train](80000/100000) average regularization: 0.04148859762214124\n",
      "[proc 1][Train] 20000 steps take 446.532 seconds\n",
      "[proc 1]sample: 66.165, forward: 220.019, backward: 51.004, update: 108.926\n",
      "[proc 0][Train] 20000 steps take 446.531 seconds\n",
      "[proc 0]sample: 67.304, forward: 213.404, backward: 49.160, update: 110.919\n",
      "[proc 1][Train](100000/100000) average pos_loss: 0.30877756952792407\n",
      "[proc 0][Train](100000/100000) average pos_loss: 0.3088074548617005\n",
      "[proc 1][Train](100000/100000) average neg_loss: 0.43929336989074946\n",
      "[proc 0][Train](100000/100000) average neg_loss: 0.43932584126144647\n",
      "[proc 1][Train](100000/100000) average loss: 0.374035469789803\n",
      "[proc 0][Train](100000/100000) average loss: 0.3740666481405497\n",
      "[proc 1][Train](100000/100000) average regularization: 0.041941663374379275\n",
      "[proc 0][Train](100000/100000) average regularization: 0.04192705822251737\n",
      "[proc 1][Train] 20000 steps take 445.965 seconds\n",
      "[proc 1]sample: 65.439, forward: 221.284, backward: 50.814, update: 107.175\n",
      "[proc 0][Train] 20000 steps take 445.964 seconds\n",
      "[proc 0]sample: 67.258, forward: 214.464, backward: 49.265, update: 110.778\n",
      "[0]Valid average MRR: 0.47699706413815457\n",
      "[0]Valid average MR: 110.01799381707251\n",
      "[0]Valid average HITS@1: 0.39202347878193605\n",
      "[0]Valid average HITS@3: 0.5097782862123441\n",
      "[0]Valid average HITS@10: 0.6398717110638993\n",
      "[proc 0]validation take 412.422 seconds:\n",
      "[1]Valid average MRR: 0.4772150236136709\n",
      "[1]Valid average MR: 108.74322299924417\n",
      "[1]Valid average HITS@1: 0.39252470089951447\n",
      "[1]Valid average HITS@3: 0.5090360010077831\n",
      "[1]Valid average HITS@10: 0.6399014006822964\n",
      "[proc 1]validation take 422.990 seconds:\n",
      "proc 1 takes 3166.468 seconds\n",
      "proc 0 takes 3166.481 seconds\n",
      "training takes 3166.9834299087524 seconds\n",
      "Save model to ckpts/DistMult_DRKG_17\n",
      "-------------- Test result --------------\n",
      "Test average MRR : 0.47610848696314323\n",
      "Test average MR : 109.2139019590486\n",
      "Test average HITS@1 : 0.39078661555118244\n",
      "Test average HITS@3 : 0.5089764192377619\n",
      "Test average HITS@10 : 0.6392374895306319\n",
      "-----------------------------------------\n",
      "testing takes 409.955 seconds\n"
     ]
    }
   ],
   "source": [
    "!DGLBACKEND=pytorch dglke_train --dataset DRKG --data_path ./dataset \\\n",
    "--data_files drkg_train.tsv drkg_valid.tsv drkg_test.tsv --format 'raw_udd_hrt' \\\n",
    "--model_name DistMult \\\n",
    "--batch_size 4096 --neg_sample_size 256 --hidden_dim 400 \\\n",
    "--gamma 200.0 --lr 0.1 --max_step 100000 -adv --regularization_coef 1.00E-07 \\\n",
    "--gpu 0 1 --num_proc 2 --mix_cpu_gpu --async_update --force_sync_interval 1000 \\\n",
    "--valid --test \\\n",
    "--batch_size_eval 128 --neg_sample_size_eval 10000 \\\n",
    "--log_interval 20000 --eval_interval 50000 --num_thread 32"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
